Ndiff ready to be tested

[Michael Pattrick]

Hey everyone,

This week I have been coding Ndiff, a utility to compare nmap xml
files and produce a XML or YAML formatted difference file. and as of
now, Ndiff is at a state of reasonable usability! So I would
appreciate it if you would try it out and tell me what you though. I'm
considering ways to remove the nmap::parser dependency but any other
feedback you could provide would be much appreciated.

You can find it here[1], it requires PERL and the latest copy of
nmap::parser[2]. The syntax is as follows:
print YAML to screen:
>ndiff.pl -y newerscan.xml older.xml
print XML to screen:
>ndiff.pl -x newerscan.xml older.xml
print XML to file:
>ndiff.pl -X outfile.xml newerscan.xml older.xml
print YAML to file:
>ndiff.pl -Y outfile.txt newerscan.xml older.xml

you can also diff multiple files at the same time:
>ndiff.pl -Y outfile.txt newestscan.xml newerscan.xml older.xml oldest.xml

I also have an up to date valid DTD and sample output XML files here [3]

Cheers,
Michael

[1] svn://svn.insecure.org/nmap-exp/ndiff
[2] http://nmapparser.wordpress.com/
[3] svn://svn.insecure.org/nmap-exp/michael/ndiff/ndiff-refrence

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org

[Fyodor]

On Thu, Jun 26, 2008 at 11:44:47PM -0400, Michael Pattrick wrote:
> Hey everyone,
>
> This week I have been coding Ndiff, a utility to compare nmap xml
> files and produce a XML or YAML formatted difference file. and as of
> now, Ndiff is at a state of reasonable usability! So I would
> appreciate it if you would try it out and tell me what you though. I'm
> considering ways to remove the nmap::parser dependency but any other
> feedback you could provide would be much appreciated.

Hi Michael.  That's great news!  I only had a little while to look at
it today (very busy), but did have some comments:

o Its great that you were able to reduce it to just 300 lines!  It is
  nice to see programs coded in a tight, efficient (yet still easy to
  read) package.

o I agree that it would be great to remove the Nmap::Parser
  dependency.  I'd guess that the vast majority of people don't have
  that installed.  Does anyone here have ideas?  If it must be
  required, maybe it can be included in some fashion.

o There is a blank space before every line, including the first
  (interpreter specifier) line.  The first 2 lines look like:

 #!/usr/bin/perl -w
 use Nmap::Parser;

My shell (probably any shell) doesn't allow that, so I have to do
"perl ndiff.pl" rather than just running ndiff.pl.  Removing those
excess spaces should do the trick.

o I'm glad you included sample xml output in the ndiff-refrence
  directory (trivial nit: it should be spelled nmap-reference), but
  can you add a yaml output example too?  It would be best if the yaml
  and xml output are comparing the same two files.

o Its great that you created a DTD too.

I hope this helps!

Cheers,
Fyodor

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org

[David Fifield]

On Thu, Jun 26, 2008 at 11:44:47PM -0400, Michael Pattrick wrote:
> This week I have been coding Ndiff, a utility to compare nmap xml
> files and produce a XML or YAML formatted difference file. and as of
> now, Ndiff is at a state of reasonable usability! So I would
> appreciate it if you would try it out and tell me what you though. I'm
> considering ways to remove the nmap::parser dependency but any other
> feedback you could provide would be much appreciated.

Great job on this. It is a challenge to start an application from
scratch and you have handled it well. My specific comments are below.

> You can find it here[1], it requires PERL and the latest copy of
> nmap::parser[2]. The syntax is as follows:
> print YAML to screen:
> >ndiff.pl -y newerscan.xml older.xml

To me the order new, old is unintuitive. I would prefer old, new like
diff takes.

> print XML to screen:
> >ndiff.pl -x newerscan.xml older.xml
> print XML to file:
> >ndiff.pl -X outfile.xml newerscan.xml older.xml
> print YAML to file:
> >ndiff.pl -Y outfile.txt newerscan.xml older.xml

I was confused about the YAML output. It seems to be serving the purpose
of plain text output. Is it really YAML? It seems like the colons in the
output would break that. It looks good for plain text, though perhaps it
could use one less level of indentation.

I can see using the plain text output the most, so I would like to see
it the default when no other output option is given. When I run without
a -x or -y option I get

Syntax: ndiff (-[y|Y]|-[x|X]) [out.file] newerscan.xml oldscan.xml [olderscan.xml] [...]

> you can also diff multiple files at the same time:
> >ndiff.pl -Y outfile.txt newestscan.xml newerscan.xml older.xml oldest.xml

That's really cool. Good idea.

> I also have an up to date valid DTD and sample output XML files here [3]

I would like to see sample input files, both as a demonstration and to
test tricky cases. You can doctor output files you generate in order to
make good samples. Here are a couple of ideas I had:
* A host with extraports in more than one state.
* A host that is identical between scans except for its address.

Packaging Nmap::Parser works for running the application from its
directory. How about installation? When ndiff.pl is installed Parser.pm
has to go somewhere unless Nmap::Parser is already installed somewhere.
Considering the small subset of the output the Ndiff uses, maybe a
simple custom parser would be appropriate. You just need one complex
enough to build a list of hosts, each with a list of services.

Can you make a version that uses a standard Perl library (XML::Writer?)
to write the XML? I'm afraid that string concatenation is too fragile.
At least you have to escape all the values you insert to get rid of
special characters like &.

Eventually Ndiff is supposed to ship with Nmap. So there is polish like
an installation mechanism that will have to be done.

The output formats and DTD are already a great contribution. It's good
that you spent adequate time developing them.

David Fifield

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org

[Fyodor]

On Tue, Jul 01, 2008 at 12:48:09AM -0600, David Fifield wrote:
> On Thu, Jun 26, 2008 at 11:44:47PM -0400, Michael Pattrick wrote:
>
> Great job on this. It is a challenge to start an application from
> scratch and you have handled it well. My specific comments are below.

Indeed!  Ndiff is looking better and better every day.

> To me the order new, old is unintuitive. I would prefer old, new like
> diff takes.

Yeah, I agree that is preferable.

> I can see using the plain text output the most, so I would like to see
> it the default when no other output option is given. When I run without
> a -x or -y option I get
>
> Syntax: ndiff (-[y|Y]|-[x|X]) [out.file] newerscan.xml oldscan.xml [olderscan.xml] [...]

I'm not convinced that basing the behavior on the caplitalization is
the best approach.  How is someone supposed to remember whether -x or
-X is the version which takes a filename?  For example, imagine
someone tries to diff three files (using your current ordering) as so:

ndiff.pl -Y newestscan.xml older.xml oldest.xml

Since the user did 'Y' instead of the proper 'y' for the situation,
does that mean newestscan.xml gets blown away?  Since the script
allows more than two files to be diff'd now, you can just catch the
problem based on too many files being listed.

Maybe allowing more than two files to be diff'd is a good idea.  I'll
reserve judgement until I see an example of what it looks like.

I wonder if '-t' for text output would be better than '-y' for Yaml?
I'm not sure that many people know what Yaml is.  Even with -t, you
could still note in the documentation that the output follows the
'yaml' format.  I haven't looked at the yaml format output yet to see
how easy it is to read, etc.

> I would like to see sample input files, both as a demonstration and to
> test tricky cases. You can doctor output files you generate in order to
> make good samples.

Yeah, that would be useful.

> Eventually Ndiff is supposed to ship with Nmap. So there is polish like
> an installation mechanism that will have to be done.

Yes.  If Ndiff proves useful enough, I think it would be good to ship
it with Nmap.  It probably won't work on Windows machines without
Perl, but that may be OK.

Cheers,
-F

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org

[Fyodor]

On Wed, Jul 02, 2008 at 07:21:55PM -0700, Fyodor wrote:

> On Tue, Jul 01, 2008 at 12:48:09AM -0600, David Fifield wrote:
> > On Thu, Jun 26, 2008 at 11:44:47PM -0400, Michael Pattrick wrote:
> >
> > Syntax: ndiff (-[y|Y]|-[x|X]) [out.file] newerscan.xml oldscan.xml [olderscan.xml] [...]
>
> I'm not convinced that basing the behavior on the caplitalization is
> the best approach.  How is someone supposed to remember whether -x or
> -X is the version which takes a filename?  For example, imagine
> someone tries to diff three files (using your current ordering) as so:
>
> ndiff.pl -Y newestscan.xml older.xml oldest.xml
>
> Since the user did 'Y' instead of the proper 'y' for the situation,
> does that mean newestscan.xml gets blown away?  Since the script
> allows more than two files to be diff'd now, you can just catch the
> problem based on too many files being listed.

I noted this potential issue, but didn't suggest any solutions.  One
idea would be to keep -x and -t (or -y) and then have a special -o
option for when you want to specify an output file.  Then you could
make -X and -T aliases for -x and -t so people don't even have to
remember the proper capitalization.

Cheers,
-F

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org

[Michael Pattrick]

On Wed, Jul 2, 2008 at 10:28 PM, Fyodor <fyodor@...> wrote:
> I noted this potential issue, but didn't suggest any solutions.  One
> idea would be to keep -x and -t (or -y) and then have a special -o
> option for when you want to specify an output file.  Then you could
> make -X and -T aliases for -x and -t so people don't even have to
> remember the proper capitalization.
>

I agree about the capitalization thing, one solution would be to use
the nmap style of output arguments -oN -oX

The advantage being that our target demographic is already familiar
with these options. This also fixes the issue of people not knowing
what YAML is, YAML can now become 'normal'.

Most people wont want xml output in their terminal anyways.

Cheers,
Michael

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org

[Arturo 'Buanzo' Busleiman]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Fyodor wrote:
| I noted this potential issue, but didn't suggest any solutions.  One
| idea would be to keep -x and -t (or -y) and then have a special -o
| option for when you want to specify an output file.  Then you could

You mean, be able to use a -o for -x, and another for -t ? Like, multiple simultaneous output saving?

- --
Arturo "Buanzo" Busleiman
Independent Security Consultant - SANS - OISSG
http://www.buanzo.com.ar/pro/eng.html
Mailing List Archives at http://archiver.mailfighter.net

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIbKkIAlpOsGhXcE0RCkcMAJ9FAGDXSsX+kkdjDL3DMgSnW08yjACfb26u
TkUkD59AlcUxDGCF2NpGHXU=
=uNLy
-----END PGP SIGNATURE-----

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org

[Fyodor]

On Thu, Jun 26, 2008 at 11:44:47PM -0400, Michael Pattrick wrote:
> Hey everyone,
>
> This week I have been coding Ndiff, a utility to compare nmap xml
> files and produce a XML or YAML formatted difference file. and as of
> now, Ndiff is at a state of reasonable usability!

I had a chance this evening to use it for a real task, which is one of
the best way to test these things.

The first thing I encountered was the lack of XML::Writer and XML:Twig
on my system.  So I tried a newer system, and that didn't have these
Perl modules either.  I was able to easily add the modules, but that
is a lot to ask of users.  Particularly since we've already lost most
of the Windows users by requiring Perl.  So it would be good for it to
work "out of the box" for as many of the remaining users as possible.

Anyway, it worked fine once I installed the modules.  I have been
testing various Nmap changes and how they affect performance and
accuracy.  So I repeated the same scan many times with different
options.  Comparing times is straightforward, but I used ndiff to test
accuracy.  It worked pretty well for this.  Though I did take some
notes for possible improvement.  Here is an representative segment of
Ndiff output.

81.174.236.66:
        Port 10000/tcp was open
        Port 10000/tcp was listening with an snet-sensor-mgmt server

It might be nice if you could list the rDNS hostname with the IP
address (if they differ, maybe just list the newer one, or maybe don't
list it in that case).  Many sysadmins recognize their machines by
hostname more than IP.

In these scans, I never used version detection.  So the lines like the
one above about snet-sensor-mgmt are of limited usefulness.  Nmap only
previously said snet-sensor-mgmt because that is what port 10,000 is
registered to in nmap-services.  Nmap XML has fields which tell how
Nmap guessed the service name.  If generated by the table, you might
want to just ignore it like you do "unknown" ports.  On the other
hand, it can be nice to remind the user what a port maps to.  Maybe
you could combine the port number and service name in one line,
something like:

  Port 10000/tcp was open (snet-sensor-mgmt)

or

  Port 49396/tcp is closed was open (snet-sensor-mgmt)

In these examples, Nmap only prints the service name next to the open
or open|filtered states, which is probably OK.  Or you could print it
in all states like:

  Port 49396/tcp is closed (snet-sensor-mgmt) was open (snet-sensor-mgmt)

It would be nice to just print the service by the port number, but
that doesn't handle the case well where the service changes due to
different version detection results.

Alowing multiple input files as an interesting idea.  But standard
diff doesn't let you do this, so I was a bit apprehensive about trying
it.  I eventually worked up the guts to do so, and it worked pretty
well.  One potential issue is that it was hard to tell what files were
being compared when I did a 3+-file diff.  Normal diff prints
filenames if there are any differences, and maybe it would be
beneficial for ndiff to do so too.

In an svn checkin you said "Ndiff now handles hosts with dynamic
addresses intelegently".  Can you describe how it does that?

I have mixed feelings about the new output options.  They are
definitely better than the ones which differed only in capitalization.
And it is nice that they are similar to Nmap's flags.  But on the
other hand, ndiff is a simpler program than Nmap and so simpler
options might benefit it.  For example, you could have options
specifying the output type (e.g. -x and -t) and then a separate option
(such as -o) to specify the filename if the user doesn't want it going
to stdout.  An issue with the current output flags is that you can't
generate XML to standard out (as far as I can tell).  You might want
to do this when you pipe the ndiff results to a program such an xml
grep utility.  Or higher level applications such as Zenmap might
prefer to read the stdout output rather than deal with a temporary
file.  If the -oN/-oX style options are kept, they should support a
'-' to mean output to standard output, just as Nmap does.  One nice
thing about -oN/-oX is that you can potentially output to both formats
at once (I haven't tested if this works).

Anyway, ndiff did the trick for me nicely!  I hope this feedback helps.

Cheers,
-F

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org

[Fyodor]

On Sat, Jul 05, 2008 at 01:23:44AM -0700, Fyodor wrote:
> On Thu, Jun 26, 2008 at 11:44:47PM -0400, Michael Pattrick wrote:
>
> 81.174.236.66:
>         Port 10000/tcp was open
>         Port 10000/tcp was listening with an snet-sensor-mgmt server

One more thing I forgot to mention is that it would be nice if it told
the new state of port 10,000 in this example.  As we've mentioned,
there are certain cases where it might not know (e.g. if some
extraports are in filtered and others are closed).  But you could
print the new port in cases like this one, where all the extraports
are in one state (filtered in this case).  Or you could extend Nmap's
xml output as has been discussed on Nmap-dev to list all the ports in
each extraports state.

Cheers,
-F


_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org

[Michael Pattrick]

Hey Fyodor,

On Sat, Jul 5, 2008 at 4:23 AM, Fyodor <fyodor@...> wrote:
> The first thing I encountered was the lack of XML::Writer and XML:Twig
> on my system.  So I tried a newer system, and that didn't have these
> Perl modules either.  I was able to easily add the modules, but that
> is a lot to ask of users.  Particularly since we've already lost most
> of the Windows users by requiring Perl.  So it would be good for it to
> work "out of the box" for as many of the remaining users as possible.

I agree, David suggested XML::writer and I happened to have it on my
system so I thought it was common, but I guess not. I could just go
back to how I did it before, concatenating strings to produce XML,
which guaranties compatibility with all. However, it concatenation
cant replace XML::twig(which I also had on my system prior to ndiff)
as it is needed to parse the xml input files. I am open to suggestions
about this, if anyone knows how this could be handled better.

> In an svn checkin you said "Ndiff now handles hosts with dynamic
> addresses intelegently".  Can you describe how it does that?

If a host with ip address A has gone offline, ndiff checks to see if
any other host with ip address B has come online which has all the
same port states, service names, and etc.

So instead of printing this:
A:
  Host has gone offline
  Port 1 was open
  Port 2 was open
  Port 3 was open

B:
  Host has come online
  Port 1 is open
  Port 2 is open
  Port 3 is open

ndiff prints:
A:
  Host has been renamed to B

I implemented this when multiple people pointed out that they have
many hosts on DHCP and were worried that the output would be cluttered
by hosts changing names.


Thanks for the input, ill try to implement it.

Cheers,
Michael

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org