We are trying to ship jira application logs via Fluent d. Need a multiline regex to parse

[Naresh Kumar Kethavath]

Below is the sample log we using :

2021-04-16 05:26:12,470-0400 localq-reader-2 ERROR [c.a.j.c.distribution.localq.LocalQCacheOpReader] Abandoning sending: LocalQCacheOp{cacheName='com.atlassian.jira.crowd.embedded.ofbiz.EagerOfBizGroupCache.groupCache', action=PUT, key={10000,ccs-cmt engineering}, value=Group[CCS-CMT Engineering:10000], creationTimeInMillis=1618503048658} from cache replication queue: [queueId=queue_ip10128170230factsetcom_1_c4cc94603bb736075782a80b0b7d53d4, queuePath=/home/svc-jira/jira/home/localq/queue_ip10128170230factsetcom_1_c4cc94603bb736075782a80b0b7d53d4], failuresCount: 10/10. Removing from queue. Error: java.rmi.UnknownHostException: Unknown host: ip-10-136-98-135.factset.com; nested exception is:
java.net.UnknownHostException: ip-10-136-98-135.factset.com
com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpSender$UnrecoverableFailure: java.rmi.UnknownHostException: Unknown host: ip-10-136-98-135.factset.com; nested exception is:
java.net.UnknownHostException: ip-10-136-98-135.factset.com
at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:88)
at com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpReader.run(LocalQCacheOpReader.java:84)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.rmi.UnknownHostException: Unknown host: ip-10-136-98-135.factset.com; nested exception is:
java.net.UnknownHostException: ip-10-136-98-135.factset.com
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:620)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342)
at com.atlassian.jira.cluster.distribution.localq.rmi.BasicRMICachePeerProvider.lookupRemoteCachePeer(BasicRMICachePeerProvider.java:64)
at com.atlassian.jira.cluster.distribution.localq.rmi.BasicRMICachePeerProvider.create(BasicRMICachePeerProvider.java:39)
at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.getCachePeerFor(CachingRMICachePeerManager.java:58)
at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.withCachePeer(CachingRMICachePeerManager.java:91)
at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:63)
... 6 more

Please find the below configuration using to parse

<source>
@type tail
@[deleted]_level debug
path /etc/td-agent/config.d/sample_log.log
pos_file /etc/td-agent/config.d/test.pos
format multiline
format_firstline /\d{4}-\d{1,2}-\d{1,2}/
format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2},\d{3})\s+(?<thread>.*)\s(?<level>(WARN|DEBUG|INFO|ERROR|FATAL))\s+(.*)?\s*\[(?<class>[^\s]*)\]\s+(?<message>.*)/
tag file.debug
</source>

<filter file.**>
@type record_transformer
<record>
__clp_index test
__clp_type aws_uat_application_jira
</record>
</filter>

## match tag=debug.** and dump to console
<match file.debug>
@type stdout
@[deleted]output_stdout
</match>

<match file.**>
@type forward
expire_dns_cache 300
dns_round_robin true
@[deleted]_level info
transport tls
keepalive true
keepalive_timeout 300s
<server>
host fluentd.log-uat.factset.com
port 24224
</server>
<buffer>
flush_interval 5s
</buffer>
</match>

Facing errors in console any better multiline regex?

[Charlie Misonne]

Hi @Naresh Kumar Kethavath

Did you manage to construct the regexes properly? We're trying to do the same.