Question Ajouter un saut de ligne entre les horodatages dans le journal avec regex


J'ai des données de journal:

2017-12-03 01:35:58 [Notice] syslog: local  IP address 
2017-12-03 01:35:58 [Notice] syslog: remote IP address 
2017-12-03 01:35:58 [Notice] syslog: primary   DNS address 
2017-12-03 01:35:58 [Notice] syslog: secondary DNS address 
2017-12-03 01:35:59 [Warning] kernel: Link State: PVC_8_0 logistic interface up.
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: started, version 2.52 cachesize 150
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: compile time options: no-IPv6 GNU-getopt no-RTC no-DBus no-I18N no-DHCP no-TFTP
2017-12-03 01:35:59 [Warning] dnsmasq[10463]: ignoring nameserver 127.0.0.1 - local interface
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: using nameserver 87.216.1.66#53(via ppp80)
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: using nameserver 87.216.1.65#53(via ppp80)
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: read /etc/hosts - 6 addresses
2017-12-03 01:36:00 [Informational] dnsmasq[10532]: started, version 2.52 cachesize 150
2017-12-03 01:36:00 [Informational] dnsmasq[10532]: compile time options: no-IPv6 GNU-getopt no-RTC no-DBus no-I18N no-DHCP no-TFTP
2017-12-03 01:36:00 [Informational] dnsmasq[10532]: using nameserver 87.216.1.66#53(via ppp80)
2017-12-03 01:36:01 [Warning] kernel: ^M
2017-12-03 01:36:01 [Warning] kernel: Send DNS Query : domain=ntp2.jazztel.com qType=A dnsServer=87.216.1.65
2017-12-03 01:36:01 [Warning] kernel: domain: ntp2.jazztel.com , IP: 87.216.1.241
2017-12-03 01:36:01 [Warning] kernel: sntp server=ntp2.jazztel.com: 0x5 ntpServerIP=87.216.1.241

Je veux ajouter un saut de ligne chaque fois que l'horodatage change pour ressembler à ceci:

2017-12-03 01:35:58 [Notice] syslog: local  IP address 
2017-12-03 01:35:58 [Notice] syslog: remote IP address 
2017-12-03 01:35:58 [Notice] syslog: primary   DNS address 
2017-12-03 01:35:58 [Notice] syslog: secondary DNS address 

2017-12-03 01:35:59 [Warning] kernel: Link State: PVC_8_0 logistic interface up.
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: started, version 2.52 cachesize 150
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: compile time options: no-IPv6 GNU-getopt no-RTC no-DBus no-I18N no-DHCP no-TFTP
2017-12-03 01:35:59 [Warning] dnsmasq[10463]: ignoring nameserver 127.0.0.1 - local interface
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: using nameserver 87.216.1.66#53(via ppp80)
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: using nameserver 87.216.1.65#53(via ppp80)
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: read /etc/hosts - 6 addresses

2017-12-03 01:36:00 [Informational] dnsmasq[10532]: started, version 2.52 cachesize 150
2017-12-03 01:36:00 [Informational] dnsmasq[10532]: compile time options: no-IPv6 GNU-getopt no-RTC no-DBus no-I18N no-DHCP no-TFTP
2017-12-03 01:36:00 [Informational] dnsmasq[10532]: using nameserver 87.216.1.66#53(via ppp80)

2017-12-03 01:36:01 [Warning] kernel: ^M
2017-12-03 01:36:01 [Warning] kernel: Send DNS Query : domain=ntp2.jazztel.com qType=A dnsServer=87.216.1.65
2017-12-03 01:36:01 [Warning] kernel: domain: ntp2.jazztel.com , IP: 87.216.1.241
2017-12-03 01:36:01 [Warning] kernel: sntp server=ntp2.jazztel.com: 0x5 ntpServerIP=87.216.1.241

Cela fonctionne à https://regexr.com:

s/(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} ).*\n(?!\1)/$0\n/g

Mais quand je l'essaie dans Terminal (OSX), il ne fait rien:

curl -s http://192.168.1.1/cgi-bin/status_log2.cgi | grep 2017 | tail -n 30 | perl -pe 's/(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} ).*\n(?!\1)/$0\n/g'

J'ai aussi essayé gsed et sed en vain.

(Bonus s'il existe un moyen de supprimer complètement tous les horodatages redondants!)


4
2017-12-03 20:48


origine




Réponses:


Pour mettre un saut de ligne avant chaque nouvelle heure:

awk '!a[$1,$2]++ && NR>1{print ""} 1'

Comment ça marche: Dans awk, $1 et $2 sont les premiers champs, dans ce cas la date et l'heure. a[$1,$2] est un tableau associatif qui compte combien de fois nous avons vu ces deux champs. Si nous avons vu cette date et heure avant, !a[$1,$2] et nous ne sommes pas sur la première ligne, NR>1, puis nous imprimons une ligne vierge pour la séparation, print "". Le final 1 est juste un raccourci pour print-the-current-line.

Exemple

Avec vos exemples de journaux dans le fichier logfile:

$ awk '!a[$1,$2]++ && NR>1{print ""} 1' logfile
2017-12-03 01:35:58 [Notice] syslog: local  IP address 
2017-12-03 01:35:58 [Notice] syslog: remote IP address 
2017-12-03 01:35:58 [Notice] syslog: primary   DNS address 
2017-12-03 01:35:58 [Notice] syslog: secondary DNS address 

2017-12-03 01:35:59 [Warning] kernel: Link State: PVC_8_0 logistic interface up.
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: started, version 2.52 cachesize 150
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: compile time options: no-IPv6 GNU-getopt no-RTC no-DBus no-I18N no-DHCP no-TFTP
2017-12-03 01:35:59 [Warning] dnsmasq[10463]: ignoring nameserver 127.0.0.1 - local interface
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: using nameserver 87.216.1.66#53(via ppp80)
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: using nameserver 87.216.1.65#53(via ppp80)
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: read /etc/hosts - 6 addresses

2017-12-03 01:36:00 [Informational] dnsmasq[10532]: started, version 2.52 cachesize 150
2017-12-03 01:36:00 [Informational] dnsmasq[10532]: compile time options: no-IPv6 GNU-getopt no-RTC no-DBus no-I18N no-DHCP no-TFTP
2017-12-03 01:36:00 [Informational] dnsmasq[10532]: using nameserver 87.216.1.66#53(via ppp80)

2017-12-03 01:36:01 [Warning] kernel: ^M
2017-12-03 01:36:01 [Warning] kernel: Send DNS Query : domain=ntp2.jazztel.com qType=A dnsServer=87.216.1.65
2017-12-03 01:36:01 [Warning] kernel: domain: ntp2.jazztel.com , IP: 87.216.1.241
2017-12-03 01:36:01 [Warning] kernel: sntp server=ntp2.jazztel.com: 0x5 ntpServerIP=87.216.1.241

Suppression des horodatages en double

$ awk '{if(a[$1,$2]++){gsub(/./," ",$1); gsub(/./," ",$2)} else if (NR>1) print""} 1' logfile
2017-12-03 01:35:58 [Notice] syslog: local  IP address 
                    [Notice] syslog: remote IP address
                    [Notice] syslog: primary DNS address
                    [Notice] syslog: secondary DNS address

2017-12-03 01:35:59 [Warning] kernel: Link State: PVC_8_0 logistic interface up.
                    [Informational] dnsmasq[10463]: started, version 2.52 cachesize 150
                    [Informational] dnsmasq[10463]: compile time options: no-IPv6 GNU-getopt no-RTC no-DBus no-I18N no-DHCP no-TFTP
                    [Warning] dnsmasq[10463]: ignoring nameserver 127.0.0.1 - local interface
                    [Informational] dnsmasq[10463]: using nameserver 87.216.1.66#53(via ppp80)
                    [Informational] dnsmasq[10463]: using nameserver 87.216.1.65#53(via ppp80)
                    [Informational] dnsmasq[10463]: read /etc/hosts - 6 addresses

2017-12-03 01:36:00 [Informational] dnsmasq[10532]: started, version 2.52 cachesize 150
                    [Informational] dnsmasq[10532]: compile time options: no-IPv6 GNU-getopt no-RTC no-DBus no-I18N no-DHCP no-TFTP
                    [Informational] dnsmasq[10532]: using nameserver 87.216.1.66#53(via ppp80)

2017-12-03 01:36:01 [Warning] kernel: ^M
                    [Warning] kernel: Send DNS Query : domain=ntp2.jazztel.com qType=A dnsServer=87.216.1.65
                    [Warning] kernel: domain: ntp2.jazztel.com , IP: 87.216.1.241
                    [Warning] kernel: sntp server=ntp2.jazztel.com: 0x5 ntpServerIP=87.216.1.241

Dans ce cas, si nous avons vu la date, $1, et le temps, $2, avant, nous remplaçons leur contenu par des blancs, gsub(/./," ",$1); gsub(/./," ",$2). Si ce n'est pas le cas et si nous ne sommes pas sur la première ligne, nous imprimons une ligne vierge pour la séparation.


2
2017-12-03 20:59



@ Ze'ev je suis content que cela ait fonctionné pour vous. Si vous êtes intéressé à en apprendre plus sur awk, un tutoriel que j'aime est le tutoriel Grymoire. - John1024


Vous devez indiquer à Perl de charger le tout immédiatement, car il lit normalement la ligne ligne par ligne. Utilisation -0777 pour ça.

Aussi, $0 dans Perl est le nom du script (-e pour un one-liner). Capturer toute la ligne et la référencer comme $1, utilisation \2 pour la date:

perl -0777 -pe 's/((\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} ).*\n)(?!\2)/$1\n/g'

2
2017-12-03 21:02





Je voudrais poster une solution (GNU) sed.

sed -nr 'h;s/^([^[]+) \[.*/\1/;x;p;:a;g;N;s/^([^\n]+)\n\1(.*)/\1\2/;Tb;:c;s/[0-9:-]([0-9 :-]+\[)/ \1/;tc;p;ba;:b;s/^[^\n]+//;P;D' logfile
2017-12-03 01:35:58 [Notice] syslog: local  IP address                                                                                      
                    [Notice] syslog: remote IP address                                                                                      
                    [Notice] syslog: primary   DNS address                                                                                  
                    [Notice] syslog: secondary DNS address                                                                                  

2017-12-03 01:35:59 [Warning] kernel: Link State: PVC_8_0 logistic interface up.                                                            
                    [Informational] dnsmasq[10463]: started, version 2.52 cachesize 150                                                     
                    [Informational] dnsmasq[10463]: compile time options: no-IPv6 GNU-getopt no-RTC no-DBus no-I18N no-DHCP no-TFTP         
                    [Warning] dnsmasq[10463]: ignoring nameserver 127.0.0.1 - local interface                                               
                    [Informational] dnsmasq[10463]: using nameserver 87.216.1.66#53(via ppp80)                                              
                    [Informational] dnsmasq[10463]: using nameserver 87.216.1.65#53(via ppp80)                                              
                    [Informational] dnsmasq[10463]: read /etc/hosts - 6 addresses                                                           

2017-12-03 01:36:00 [Informational] dnsmasq[10532]: started, version 2.52 cachesize 150                                                     
                    [Informational] dnsmasq[10532]: compile time options: no-IPv6 GNU-getopt no-RTC no-DBus no-I18N no-DHCP no-TFTP         
                    [Informational] dnsmasq[10532]: using nameserver 87.216.1.66#53(via ppp80)                                              

2017-12-03 01:36:01 [Warning] kernel: ^M                                                                                                    
                    [Warning] kernel: Send DNS Query : domain=ntp2.jazztel.com qType=A dnsServer=87.216.1.65
                    [Warning] kernel: domain: ntp2.jazztel.com , IP: 87.216.1.241
                    [Warning] kernel: sntp server=ntp2.jazztel.com: 0x5 ntpServerIP=87.216.1.241

1
2017-12-03 23:31