Goldengate Troubleshooting

Hi ,

General errors and solutions for goldengate operations

1-)
After goldengate upgrade from 12c to 19c , mapping error for replicat.
You should add keycols parameter to map statement .
ACCOUNT_NO is uniq index or primary key.
OGG-01296 Error mapping from ARMOR.CMF to ARMOR.CMF.
MAP ARMOR.CMF, TARGET ARMOR.CMF, KEYCOLS (ACCOUNT_NO);

2-)
If there is replicat checkpoint error problem , goldengate table locks should be checked.
select /*+parallel(32)*/ * from gv$session where username='GOLDENGATE_ADMIN' and module like '%REP15%';
select /*+parallel(32)*/ distinct 'alter system kill session '''||f.sid||','||f.serial#||',@'||f.INST_ID||''' immediate ;' kll_ss from gv$session f where inst_id=8 and sid in (4162,574);
select /*+parallel(32)*/ * from gv$sqlarea where sql_id='2wh9qssuhu01a';

3-)
Goldengate report file must be carefully analyzed. "Caused by" is clue data.
cat RHDFGN14.rpt | grep "Caused by"
Caused by: java.io.IOException: HDFS file [/data/domain/billing/armor/armor.cdr_data] cannot be created.  File already exists.

4-)
"OGG-01169" , add KEYCOLS to map statement on replicat .
ERROR OGG-01169 Encountered an update where all key columns for target table ODS.SBL_S_ORDER_X are not present.
MAP SIEBEL.S_ORDER_X, TARGET ODS.SBL_S_ORDER_X, KEYCOLS (ROW_ID);
  
5-) 
"OGG-06593" , pump abended problem , etrollover should be used.
alter extract PBDATA,etrollover
start PBDATA 

6-)
“OGG-01163” Bad column length (239) specified for column PARAMETER_VALUE
Table causing the problem was commented and started.
Table has 2000 rows.

#target select count(*) from ARMOR.CCB_PACKAGE_PARAMETER@SOURCE_DB; select count(*) from ARMOR.CCB_PACKAGE_PARAMETER; #target create table ARMOR.CCB_PACKAGE_PARAMETER_ydk as select * from ARMOR.CCB_PACKAGE_PARAMETER; truncate table ARMOR.CCB_PACKAGE_PARAMETER;
#target
Table truncate and all insert or minux insert data . You select.
select * from ARMOR.CCB_PACKAGE_PARAMETER@SOURCE_DB
minus
select * from ARMOR.CCB_PACKAGE_PARAMETER;

#target insert into ARMOR.CCB_PACKAGE_PARAMETER select * from ARMOR.CCB_PACKAGE_PARAMETER@SOURCE_DB; commit; #target goldengate #Table causing the problem was uncommented and started. stop R01ARMOR edit param R01ARMOR start R01ARMOR info R01ARMOR

7-)
Oracle db has “enq: TX – allocate ITL entry” event for goldengate tables .
You increase initrans and pctfree .

select /*+parallel(32)*/ event,count(*) from gv$active_session_history WHERE user_id=488 and SAMPLE_TIME>sysdate-1/24/4 group by event order by 2 desc;
--enq: TX - allocate ITL entry

select /*+parallel(32)*/ * from gv$active_session_history;

select /*+parallel(32)*/ CURRENT_OBJ# from gv$active_session_history where event='enq: TX - allocate ITL entry' and SAMPLE_TIME>sysdate-1/24/4;

select /*+parallel(32)*/ * from dba_objects where OBJECT_ID=253061706;

select /*+parallel(32)*/ * from ODS.SBL_S_ORDER_ITEM partition(SYS_P32947295);
select /*+parallel(32)*/ * from dba_tab_partitions where partition_name='SYS_P32947295';
--TO_DATE(' 2019-09-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

select /*+parallel(32)*/ * from dba_users where username like '%GOLDENGATE_ADMIN%';
--GOLDENGATE_ADMIN

--increase
alter table <name> pctfree 50;
alter table <name> initrans 100;

8-)
Goldengate disk full problem.
You find out which files fill disk.
ls -lrt /goldengate_home_path/dirdat/sblpro/
You add following statement to manager param file and restart mgr.
PURGEOLDEXTRACTS ./dirdat/sblpro/dh*, USECHECKPOINTS,MINKEEPDAYS 1
stop mgr 
y
start mgr 

9-)
For solution , delete add pump
2019-07-23 09:10:25  ERROR   OGG-06601  Mismatch between the length of seqno from checkpoint (9) and recovery (6) for extract trail ./dirdat/sxlpro/sb.

10-)
Pump problem , you control target disk .
2019-07-23 09:25:10 ERROR OGG-01031 There is a problem in network communication, a remote file problem, encryption keys for target and source do not match (if using ENCRYPT) or an unknown error. (Reply received is Could not flush “.
/dirdat/sblpro/sb000014166” (error 28, No space left on device)).

11-)
Mgr already listened port problem , you kill it. 7813 is mgr port that is used for our mgr port.
ERROR OGG-01224 Address already in use.
netstat -nap | grep 7813
tcp        0      0 :::7810                     :::*                        LISTEN      7590/./mgr
kill -9 7590

12-)
Replicat is done etrollover . If problem continue , you do initial load for table.
ERROR OGG-01061 Invalid CSN value length(0) from data source during recovery

13-)
You should create directories with mkdir on target as in GLOBALS parameter file dest strings.
ERROR OGG-01224 TCP/IP error 111 (Connection refused), endpoint: 10.86.85.29:7809.

14-)
Server process error for goldengate threads.
fork() failed creating new process
You should increase ggate server user nproc value .You kill ggate process and restart goldengate.

15-)

You should solve for this medhot.
SQL> @/ggate2/ggate12c/prvtlmpg.plb;
2019-05-14 08:01:47 ERROR OGG-02912 Patch 17030189 is required on your Oracle mining database for trail format RELEASE 12.2 or later.

16-) Pump etrollover use.
2019-05-14 08:22:21 ERROR OGG-01411 Cannot convert input file ./dirdat/bigdata02/kn000000000 with format RELEASE 12.1 to output file /evam/ggs_adapters_122/dirdat/KNCpro2_new/k2000000 with format RELEASE 12.2.
alter PBDATA_2 etrollover

17-)
You add RMTTRAIL for pump. 
2019-05-14 08:37:53  ERROR   OGG-01044  The trail '/ggatea01/KNCpro1_122c/k1' is not assigned to extract 'PKNC01_1'. Assign the trail to the extract with the command "ADD EXTTRAIL/RMTTRAIL /ggatea01/KNCpro1_122c/k1, EXTRACT PKNC01_1".
GGSCI (kncrbdp01.vodafone.local) 35> ADD RMTTRAIL /ggatea01/KNCpro1_122c/k1 EXTRACT PKNC01_1, MEGABYTES 600

18-)
You should use PASSTHRU for this problem.
2019-05-14 09:57:05  ERROR   OGG-03533  Conversion from character set UTF-8 of source column DEVICE_MODEL to character set we8iso8859p9 of target column DEVICE_MODEL failed because the source column contains a character 'e2 80 9d' at of
fset 2 that is not available in the target character set.
MAP VFTRPC.PC_OFFER_DETAILS, TARGET WEBDBMC.PC_OFFER_DETAILS, COLCHARSET( PASSTHRU,DISP_NAME_TRK,PROD_NAME,COMM_PROD_NAME,DEVICE_MODEL);

19-)
You restore archived logs on source database.
2019-05-10 09:56:30 ERROR OGG-00868 Error code 1291, error message: ORA-01291: missing logfile
(Missing Log File WAITING FOR DICTIONARY REDO. Read Position SCN: 3751.45909845 (16110468237141)).

20-)
Goldengate pump take different port , OGG-01232 Receive TCP params error: TCP/IP error 104
alter extract PBDATA_5,etrollover
refresh mgr
start PBDATA_5
info PBDATA_5

21-)
You alter table as null .
#target goldengate
Error=“Error ORA-01400: cannot insert NULL into”
POST.BCA_PYMNT_DATA to POST.BCA_PYMNT_DATA

#target db
alter TABLE KNNPRO4_POST_BILL.BCA_PAYMENT_DATA modify BILL_GROUP_NO NULL;

22-)
BATCHSQL commented and uncommented and started. You should solve with this method.
2019-04-13 07:08:54 WARNING OGG-00869 Aborting BATCHSQL transaction. Database error 600 (ORA-00600: internal error code, arguments: [ktecgetsh-inc], [1], [], [], [],