[flow-tools] Start time and end time of the processed raw data
Mark Fullmer
maf@splintered.net
Tue, 30 Apr 2002 23:23:39 -0400
On Thu, Apr 25, 2002 at 11:44:31AM -0700, Annie Tong wrote:
> Thanks Mark! Do I have to call ftio_write_header() to write the time
> info to the stream header?
Try this, it adds a -k flag to flow-filter which will attempt to
keep the start end end time.
Index: flow-filter.c
===================================================================
RCS file: /usr/home/djnz-cvsroot/flow-tools/src/flow-filter.c,v
retrieving revision 1.23
diff -c -r1.23 flow-filter.c
*** flow-filter.c 2002/04/19 21:08:48 1.23
--- flow-filter.c 2002/05/01 02:04:38
***************
*** 73,78 ****
--- 73,79 ----
struct ftset ftset;
struct ftver ftv;
struct ftprof ftp;
+ u_int32 time_start, time_end;
int i, ret;
char *acl_fname, *acl_std_src_name, *acl_std_dst_name;
char *acl_ext_name, *str, *strm;
***************
*** 80,85 ****
--- 81,87 ----
int acl_std_dst_index, acl_std_dst_index2;
int acl_ext_index, acl_ext_index2;
struct acl_ip_ext_entry tmp_ext;
+ int keep_input_time;
int filter_input, filter_output, filter_srcport, filter_dstport;
int filter_prot, filter_srcas, filter_dstas, filter_tos, filter_tcp_flags;
char in_tbl[65536], out_tbl[65536], src_tbl[65536], dst_tbl[65536];
***************
*** 112,123 ****
total_flows = 0;
tos_mask = 0xff;
tcp_flags_mask = 0xff;
filter_input = filter_output = filter_srcport = filter_dstport = 0;
filter_prot = filter_srcas = filter_dstas = filter_tos = 0;
filter_tcp_flags = 0;
! while ((i = getopt(argc, argv, "a:A:b:C:d:f:p:P:r:S:t:T:D:E:z:i:I:o")) != -1)
switch (i) {
case 'a': /* src AS filter list */
--- 114,126 ----
total_flows = 0;
tos_mask = 0xff;
tcp_flags_mask = 0xff;
+ keep_input_time = 0;
filter_input = filter_output = filter_srcport = filter_dstport = 0;
filter_prot = filter_srcas = filter_dstas = filter_tos = 0;
filter_tcp_flags = 0;
! while ((i = getopt(argc, argv, "a:A:b:C:d:f:kp:P:r:S:t:T:D:E:z:i:I:o")) != -1)
switch (i) {
case 'a': /* src AS filter list */
***************
*** 173,178 ****
--- 176,186 ----
filter_output = 1;
break;
+ case 'k': /* keep the start/end time from the input */
+
+ keep_input_time = 1;
+ break;
+
case 'P': /* filter dstport */
if (load_lookup(optarg, 65536, dst_tbl))
***************
*** 313,318 ****
--- 321,341 ----
((ftset.z_level) ? FT_IO_FLAG_ZINIT : 0) ) < 0)
fterr_errx(1, "ftio_init(): failed");
+ /* preserve start/end time from input stream? */
+ if (keep_input_time) {
+
+ time_start = ftio_get_cap_start(&ftio_in);
+ time_end = ftio_get_cap_end(&ftio_in);
+
+ if (time_start && time_end) {
+
+ ftio_set_preloaded(&ftio_out, 1);
+ ftio_set_cap_time(&ftio_out, time_start, time_end);
+
+ }
+
+ }
+
ftio_set_comment(&ftio_out, ftset.comments);
ftio_set_byte_order(&ftio_out, ftset.byte_order);
ftio_set_z_level(&ftio_out, ftset.z_level);
***************
*** 561,567 ****
void usage(void) {
! fprintf(stderr, "Usage: flow-filter [-ho] [-a src_as_filter] [-A dst_as_filter] [-b big|little]\n");
fprintf(stderr, " [-C comment] [-D dstaddr_filter_name] [-d debug_level] [-f acl_fname]\n");
fprintf(stderr, " [-i input_filter] [-I output_filter] [-p srcport_filter]\n");
fprintf(stderr, " [-P dstport_filter] [-r ipprot_filter] [-S srcaddr_filter_name]\n");
--- 584,590 ----
void usage(void) {
! fprintf(stderr, "Usage: flow-filter [-hko] [-a src_as_filter] [-A dst_as_filter] [-b big|little]\n");
fprintf(stderr, " [-C comment] [-D dstaddr_filter_name] [-d debug_level] [-f acl_fname]\n");
fprintf(stderr, " [-i input_filter] [-I output_filter] [-p srcport_filter]\n");
fprintf(stderr, " [-P dstport_filter] [-r ipprot_filter] [-S srcaddr_filter_name]\n");
>
> Regards,
>
> Annie Tong
> MAE Engineering
> MCI WorldCom
>
> Mark Fullmer wrote:
>
> >The problem is some of the information flow-capture emits in the header can
> >not be known until after processing the entire stream. For example
> >the number of flows may change after filtering. When the output is
> >pipe there's no way to rewind and fix the header.
> >
> >flow-cat has a -p flag which will preload all the headers and compute
> >start/end, nflows, etc. I guess flow-filter and friends could at least
> >preserve the time...
> >
> >It's not that difficult to fix. All that needs to be done in is something
> >like:
> >
> >time_start = ftio_get_cap_start(&ftio_in);
> >time_end = ftio_get_cap_end(&ftio_in);
> >ftio_set_cap_time(&ftio_out, time_start, time_end);
> >
> >This will be the header time, not the time of the first non filtered
> >flow though.
> >
> >mark
> >
> >On Mon, Apr 22, 2002 at 01:20:25PM -0700, Annie Tong wrote:
> >
> >>Thanks Mark! You mentioned that the headers info will be gone if the
> >>flows have been processed, is there any way that I can preserve the
> >>header info in the processed flow? I passed the -p flag to flow-stat to
> >>generate report on data that has been processed by flow-filter, it
> >>prints "note, incomplete flow file" in the report. From ftio.c, it
> >>seems the flag FT_HEADER_FLAG_PRELOADED is reset to 0 after the data has
> >>been processed, how can I set the flag back to 1?
> >>
> >>Thank you.
> >>
> >>Regards,
> >>
> >>Annie Tong
> >>MAE Engineering
> >>MCI WorldCom
> >>
> >>
> >>
> >>Mark Fullmer wrote:
> >>
> >>>Passing the -p flag to flow-stat will print additional header
> >>>information, ie
> >>>
> >>># mode: normal
> >>># capture hostname: XXXX
> >>># exporter IP address: X.X.X.X
> >>># capture start: Sat Apr 20 12:45:00 2002
> >>># capture end: Sat Apr 20 12:50:00 2002
> >>># capture period: 300 seconds
> >>># compress: on
> >>># byte order: little
> >>># stream version: 3
> >>># export version: 5
> >>># lost flows: 0
> >>># corrupt packets: 0
> >>># sequencer resets: 0
> >>># capture flows: 123244
> >>>
> >>>Unfortunately if the flows have been processed the headers from
> >>>flow-capture are
> >>>usually gone, in which case you would need to compute them on the fly.
> >>>
> >>>See ftio_header_print() in ftio.c and flow-print.c for more details.
> >>>
> >>>mark
> >>>
> >>>On Fri, Apr 19, 2002 at 01:47:52PM -0700, Annie Tong wrote:
> >>>
> >>>>Hi Mark,
> >>>>
> >>>>I'm trying to add the duration of the processed raw data in the header
> >>>>of the report that is generated by flow-stat in the following format,
> >>>>
> >>>>"Processed <number of flows> flows between <Day> <Month> <Date> <Year>
> >>>><Time> and <Day> <Month> <Date> <Year> <Time>"
> >>>>
> >>>>e.g.
> >>>>"Processed 20000 flows between Fri Apr 12 2002 00:00:00 and Fri Apr 12
> >>>>2002 09:59:59"
> >>>>
> >>>>I'm looking at your code flow-stat.c and found 2 variables,
> >>>>fs0.time_start and fs0.time_end, which stores the start_time and the
> >>>>end_time of the processed raw data, and they're in unsigned integer.
> >>>>Can I use the function localtime() to convert them in the format I
> >>>>want? Also where did you get the start_time and end_time of the
> >>>>processed data? Are they stored in the packet as $startime and
> >>>>$endtime (found the reference from Cflow.pm)?
> >>>>
> >>>>Thank you!
> >>>>
> >>>>Regards,
> >>>>
> >>>>Annie Tong
> >>>>MAE Engineering
> >>>>MCI WorldCom
> >>>>
> >>>>
> >>>>_______________________________________________
> >>>>flow-tools@splintered.net
> >>>>http://www.splintered.net/sw/flow-tools
> >>>>
> >>>_______________________________________________
> >>>flow-tools@splintered.net
> >>>http://www.splintered.net/sw/flow-tools
> >>>
> >
>