Cloudera CDH/CDP 및 Hadoop EcoSystem, Semantic IoT등의 개발/운영 기술을 정리합니다. gooper@gooper.com로 문의 주세요.
impala [CDP7.1.7]impala-shell을 이용하여 kudu table에 insert/update수행시 발생하는 오류(Transport endpoint is not connected (error 107)) 발생시 확인할 내용
오류 메세지 : Transport endpoint is not connected (error 107)
원인 : 특정 Kudu테이블에 동시에 혹은 연속적으로 많은 update/insert쿼리가 수행되는 경우
조치 : Kudu tablet server의 configuration에서 --rpc_max_message_size를 200MB 혹은 250MB정도로 증가 시키고 서비스를 restart해주면 오류를 해소 혹은 완화 할 수 있다.
======================== Coordinator 역할 impalad log file ========
node_idcp01.gooper.com INFO November 21, 2023 10:50 AM cc:678
ExecState: query id=11454a6d71dc7db7:7201b9e000000000 finstance=11454a6d71dc7db7:7201b9e000000013 on host=node_datp09.gooper.com:22000 (EXECUTING -> ERROR) status=Unable to advance iterator for node with id '0' for Kudu table 'impala::db_name.table_name': Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
======================== Executor 약할의 impalad log file===========
10:20:08.959 AM INFO cc:306
9a497afbb0dcd55d:7ab21eed00000448] DeregisterRecvr(): fragment_instance_id=9a497afbb0dcd55d:7ab21eed00000448, node=241
10:20:09.011 AM INFO cc:337
9a497afbb0dcd55d:7ab21eed00000448] cancelling active streams for fragment_instance_id=9a497afbb0dcd55d:7ab21eed00000448
10:20:09.011 AM INFO cc:734
9a497afbb0dcd55d:7ab21eed00000448] Instance completed. instance_id=9a497afbb0dcd55d:7ab21eed00000448 #in-flight=56 status=OK
10:20:09.112 AM INFO cc:306
9a497afbb0dcd55d:7ab21eed00000459] DeregisterRecvr(): fragment_instance_id=9a497afbb0dcd55d:7ab21eed00000459, node=243
10:20:15.563 AM INFO cc:72
61400b1a312547d7:e3cc9f570000000a] Serialized kudu.tserver.ScanRequestPB (134219283 bytes) is larger than the maximum configured RPC message size (52428800 bytes). Sending anyway, but peer may reject the data.
10:20:15.641 AM INFO cc:664
client connection to 11.111.111.39:7050 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
10:20:15.641 AM INFO cc:1337
f44fa39abb1a8044:2ddb9a720000000b] marking tablet server 52bf6df6c61a442c8847fbd228672bc9 (node_datp19.gooper.com:7050) as failed
10:20:15.641 AM INFO cc:1337
61400b1a312547d7:e3cc9f570000000a] marking tablet server 52bf6df6c61a442c8847fbd228672bc9 (node_datp19.gooper.com:7050) as failed
10:20:15.641 AM INFO cc:288
f44fa39abb1a8044:2ddb9a720000000b] tablet 4561250e99ea415c8adf1ed18f4eefcc: replica 52bf6df6c61a442c8847fbd228672bc9 (node_datp19.gooper.com:7050) has failed: Network error: TS failed: recv error from unknown peer: Transport endpoint is not connected (error 107)
10:20:15.659 AM INFO cc:72
61400b1a312547d7:e3cc9f570000000a] Serialized kudu.tserver.ScanRequestPB (134219283 bytes) is larger than the maximum configured RPC message size (52428800 bytes). Sending anyway, but peer may reject the data.
10:20:15.653 AM INFO cc:129
f44fa39abb1a8044:2ddb9a720000000b] Unable to advance iterator for node with id '0' for Kudu table 'impala::db_name.table_name': Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
@ 0xc1fe33
@ 0x1620fa1
@ 0x1624595
@ 0x15878a9
@ 0x15891f8
@ 0x144f531
@ 0x14509aa
@ 0x1ca1101
@ 0x7f512a6c62dd
@ 0x7f5127183e82
===================== kudu_tserver log file 내용 =====================
628292 I1121 10:20:13.968837 3471434 maintenance_manager.cc:382] P 9d84c6e1acd245f99d242449521b64cc: Scheduling UndoDeltaBlockGCOp(07db2efadf1b47a6936aaf0808de1339): 51660 bytes on disk
628293 I1121 10:20:13.969128 3471142 maintenance_manager.cc:603] P 9d84c6e1acd245f99d242449521b64cc: UndoDeltaBlockGCOp(07db2efadf1b47a6936aaf0808de1339) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":4,"lbm_read_time_us":45,"lbm_reads_lt_1ms":16}
628294 I1121 10:20:14.153570 3471434 maintenance_manager.cc:382] P 9d84c6e1acd245f99d242449521b64cc: Scheduling CompactRowSetsOp(3275654fd0a8404fa28d667d664135a3): perf score=0.205744
628295 I1121 10:20:14.853590 3471141 maintenance_manager.cc:603] P 9d84c6e1acd245f99d242449521b64cc: CompactRowSetsOp(07db2efadf1b47a6936aaf0808de1339) complete. Timing: real 4.044s user 3.590s sys 0.290s Metrics: {"bytes_written":154984322,"cfile_cache_hit":392,"cfile_cache_hit_bytes":14304480,"cfile_cache_miss":2608,"cfile_cache_miss_bytes":257784062,"cfile_init":200,"delta_iterators_relevant":8,"dirs.queue_time_us":342915,"dirs.run_cpu_time_us":33830,"d irs.run_wall_time_us":37009,"drs_written":4,"fdatasync":127,"fdatasync_us":51589,"lbm_read_time_us":111171,"lbm_reads_lt_1ms":3408,"lbm_write_time_us":102309,"lbm_writes_lt_1ms":10049,"mutex_wait_us":18,"num_input_rowsets":4,"rows_written":859031,"spinlock_wait_cycles":56704,"thread_start_us":777,"threads_started":25,"wal-append.queue_time_us":71}
628296 I1121 10:20:15.031082 3471434 maintenance_manager.cc:382] P 9d84c6e1acd245f99d242449521b64cc: Scheduling CompactRowSetsOp(3275654fd0a8404fa28d667d664135a3): perf score=0.195746
628297 W1121 10:20:16.467859 3471134 connection.cc:664] server connection from 11.111.111.35:36948 recv error: Network error: RPC frame had a length of 134219341, but we only support messages up to 134217728 bytes long.
628298 W1121 10:20:16.467888 3471134 connection.cc:295] Shutting down server connection from 11.111.111.35:36948 with pending inbound data (4/134219341 bytes received, last active 0 ns ago, status=Network error: RPC frame had a length of 134219341, but we only support messages up to 134217728 bytes long.)
628299 W1121 10:20:16.559211 3471137 connection.cc:664] server connection from 11.111.111.33:51706 recv error: Network error: RPC frame had a length of 134219341, but we only support messages up to 134217728 bytes long.
628300 W1121 10:20:16.559245 3471137 connection.cc:295] Shutting down server connection from 11.111.111.33:51706 with pending inbound data (4/134219341 bytes received, last active 0 ns ago, status=Network error: RPC frame had a length of 134219341, but we only support messages up to 134217728 bytes long.)
628301 W1121 10:20:17.182857 3471134 connection.cc:664] server connection from 11.111.111.35:38176 recv error: Network error: RPC frame had a length of 134219340, but we only support messages up to 134217728 bytes long.
628302 W1121 10:20:17.182881 3471134 connection.cc:295] Shutting down server connection from 11.111.111.35:38176 with pending inbound data (4/134219340 bytes received, last active 0 ns ago, status=Network error: RPC frame had a length of 134219340, but we only support messages up to 134217728 bytes long.)
628303 W1121 10:20:17.292801 3471134 connection.cc:664] server connection from 11.111.111.33:52788 recv error: Network error: RPC frame had a length of 134219340, but we only support messages up to 134217728 bytes long.
628304 W1121 10:20:17.292827 3471134 connection.cc:295] Shutting down server connection from 11.111.111.33:52788 with pending inbound data (4/134219340 bytes received, last active 0 ns ago, status=Network error: RPC frame had a length of 134219340, but we only support messages up to 134217728 bytes long.)
628305 I1121 10:20:17.481701 3471139 maintenance_manager.cc:603] P 9d84c6e1acd245f99d242449521b64cc: CompactRowSetsOp(eb28422960104df2a553161f4385d726) complete. Timing: real 18.869s user 6.304s sys 0.412s Metrics: {"bytes_written": 40193385,"cfile_cache_hit":362,"cfile_cache_hit_bytes":4987939,"cfile_cache_miss":4974,"cfile_cache_miss_bytes":217966365,"cfile_init":208,"delta_iterators_relevant":4,"dirs.queue_time_us":5963,"dirs.run_cpu_time_us":4649,"dirs.run_wall_time_us":6201,"drs_written":4,"fdatasync":123,"fdatasync_us":12074,"lbm_read_time_us":11465204,"lbm_reads_1-10_ms":109,"lbm_reads_10-100_ms":209,"lbm_reads_gt_100_ms":21,"lbm_reads_lt_1ms":5467,"lbm_write_time_us":152850,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":23230,"mutex_wait_us":382,"num_input_rowsets":4,"rows_written":2189548,"spinlock_wait_cycles":85888,"thread_start_us":893,"threads_started":28,"wal-append.queue_time_us":75}