huge speed-up on filescan: with sources & examples

I can only speak for the Linux implementation, but I notice it takes some time to scan through all the files and check for updates. My favorite IDE IntelliJ IDEA has a cross-platform solution for this problem as they also need to be aware of changes on the disk. They have a tool called "fsnotifier" available under the Apache 2.0 license:

http://git.jetbrains.org/?p=idea/community.git;a=tree;f=native/fsNotifier;hb=HEAD

You feed it via stdin with the directory-roots you want it to watch and it tells you via stdout when files are created, deleted or changed. For example if I want to watch the directories /media/audio and /media/video I'd run fsnotifier and send this via stdin:

  Code:

ROOTS
/media/audio
/media/video
#

That's it. I then get a reply on stdout of what directories are unwatchable on my OS/filesystem:

  Code:

UNWATCHEABLE
/proc
/sys
/sys/fs/fuse/connections
/sys/kernel/debug
/sys/kernel/security
/dev
/dev/pts
/proc/sys/fs/binfmt_misc
/home/cnsmrl/.gvfs
#

As soon as a new file appears this gets out:

  Code:

CREATE
/media/video/Better of Ted - S03E01 - Rebirth.mp4
CHANGE
/media/video/Better of Ted - S03E01 - Rebirth.mp4
CHANGE
/media/video/Better of Ted - S03E01 - Rebirth.mp4

And when a file is changed or deleted it does look pretty similar - I'm sure you get it. Here's the source of IntelliJ IDEA where fsnotifier is actually used on different platforms with fallback if there's no fsnotifier for that platform:

http://git.jetbrains.org/?p=idea/community.git;a=blob;f=platform/platform-impl/src/com/intellij/openapi/vfs/impl/local/FileWatcher.java;hb=HEAD

There's an implementation for Linux & MacOS as well as a 3rd party BSD implementation https://github.com/skirge/fsnotifier-freebsd . I'm not sure where to find the source code for the Windows executable. But there are pre-compiled binaries available from Jetbrains for Linux, Mac & Windows: http://git.jetbrains.org/?p=idea/community.git;a=tree;f=bin;hb=HEAD

[zip]

thanks. I'll implement something similar for after 1.0. The problem is not really finding the new files, it's the cataloging process / metadata retrieval that takes the most time.