I bought a license simply beacuse Sublime Text was the only editor that could open a wikipedia json data dump. I installed many editors before I more than willingly paid for Sublime.
For me, same thing: no other editor gives me the decent UI of a GUI-based editor, but with the speed of vim when working with large text files (e.g. 10MB+).
I will often use sed or other CLI tools to parse through giant files (1GB+), but sublime only takes a short bit to load in a pretty hefty file, or a do regex search across entire projects with multiple-GB of files, and it works natively on Mac and Windows, so even when I'm forced to work on something on my Windows 10 laptop, I can be productive.
I'm glad I posted here because one point is maybe finally clear to me about speed. It's not a good word for performance. It's the wrong word to use.
The great folks at Elasticsearch would _love_ for Lucene to be more performant. It would make life so much easier for them.
The Lucene team spend a good buck on their nightly performance tests. It's astonishing how well-tested Lucene is.
I wonder why I'm faster at writing and reading. Maybe it's because I have been benchmarking against an older version (4.8). But still. I wonder if my tests are all wrong or if I just got lucky in my design. ResinDB has flaws. It puts massive pressure on GC at writing time, if your batches are huge. I'm working hard at optimizing that achilees heel away. Before I have completely done so, writing speed is achieved through lots of memory allocations. It's surprisingly easy though to move away from using GC as a service.
Not at all an unrelated issue to me, but unresolvable at the moment me thinks. To use Resin within the same process as a Go app Resin would have to be a Go library.
I'm glad you posted this because I don't think there is a embedded search engine library for Go, which is both a little funny but could also constitute a business optortunity for a Go programmer.
Would you care to talk a little more about your requirements?
Sure. I'm using it for a mildly distributed locally focused offline-able content addressable store _(that's a mouthful)_. Think Camlistore, with the things that I wanted. Personal storage is the main use case but with some limited database capabilities.
As such, the records stored in the .. store, need to be indexed with provided fields for later retrieval. The indexer is responsible for this. This ends up being far more like a "database" than anything, honestly, as my queries can be complex, or simple. Eg, tags:foo title:hasWord:bar, etc.
So basically the indexer should be able to run a full suite of database-like queries, I just don't care about the data being retrieved, only the id(s) that matches the queries. To reiterate, the indexer is just responsible for returning the content hashes/ids. The content addressed store actually stores/retrieves the data. Needed operations are all the standard ones: AND, EQ, OR, NOT, PREFIX/SUFFIX is nice too but not required, etc. and of course FullTextSearch.
"Groonga is an open-source fulltext search engine and column store."
We seem to be at least cousins. Thx for that link. I will have to get back to you.
Edit:
Groonga seems to be cloud software. ResinDB is a in-process library, not a service.
Put ResinDB behind a service end-point and you have "ResinDB as a service", much more like the Groonga architecture.
Orchestration of read/write in a distributed service-like environment is something that is not solved within the ResinDB codebase. ResinDB is intended to be a component of a distributed database, not a distributed database in itself.
Groonga has been around since 2011. I started on ResinDB last year, in March of 2016.
Groonga make monthly releases. I take long pauses because of my lifestyle.
Groonga is a team of devs. I'm an independent solo dev.
Groonga is unmanaged code. ResinDB is managed code.
You could use apiblueprint.org and swagger.io to create SDK bindings in various languages for your distributed search engine service. Which you can build using a Paxos library for the consensus algorithm, (lib)torrent for the data-exchange and the s2n or openssl library for SSL/TLS encryption.
# User facing values
None of your points in the last four paragraphs, even if impressive from a developer angle, are of any relevance to a paying customer (end-user). Except your end-user is thrilled and motivated like you are. But even then, you need to keep the motivation up with excellent and enjoyable docs, tutorial a cool website and good integration into developer tools.
# Growth Hacking
After reading the whole discussion like I've the impression that you're looking for growth hacking, but have no idea how to express it other than with differntiating features. Marketing and growth hacking is really different in that it doesn't exploit clean-ness, but messy-ness. That means your whole taks as a growth hacker/marketer is to convince a (healthily) growing mass of people, decision-makers and early-adopters using manipulative tricks. Be it neuro-marketing, selling-techniques, (programmatic) scaling at and taking an advantage or any other form of gaining mass-recognition and presence. You can find a more concise and useful explanation of this on your digital book-shelf.
Summarize how it changed my belief system (or gave it a dent)? Or summarize the video? Ok, here's a summary: the video is just one of the most watchable things I've come across.
That is incorrect. He first mentions CIA fronts, but then goes on to discuss real companies that occasionally do business with the CIA 'on the side'. Two of these (Sportsflight and Richmor) sued each others on activities that happened to be tied to the CIA.
He did not make that claim out of thin air, but produced documents that were publicly available.
The New Yorker provided more details on this topic in a piece they did on Paglen:
"More than fifteen hundred documents were made public in the case: invoices, e-mails, cell-phone logs, receipts. These contained financial data—a typical rendition itinerary seems to have cost the government about three hundred thousand dollars—and structural details of the rendition program."
Thx for the feedback. What I think you should and hope you already do realise is Lucene is nowhere near maximum performance for full-text search nor is it's relevance. And implementing new scoring routines is a drag in Lucene.
Google is also nowhere near maximum relevance. I like word2vec. That model fits into my world view. I'm going to implement it and then take it further. Hopefully while being funded. If not then it shall be my contribution to the open source space and nothing more.
If you want to do word vector similarity search, try the "annoy" library from Spotify. It's much much faster than Gensim. https://github.com/spotify/annoy
Me personally I don't think it is impressing of Google to be able to store every web page in existance and to refresh them every other minute. I just don't think that is a good way of spending electricity while we haven't figured out yet how to properly utilize the sun's energy. Google is all-knowing while burning shit-loads of coal. What's impressive about that?
My claims are backed up by the code I've spent blood and sweat to create. Disprove me please because I need to know of scenarios that I need to solve that goes into vNext, scenarios were I'm currently not doing great.
Edit: and also: I'm reaching out to you guys not because I want a pat on the back or free PR. I'm looking for advise as to how to move from having unique tech to having a business. Is this the right forum?
That's not how it works. Why would I waste my time testing your software when you don't seem to possess common sense or experience? The probability that you can back up what you say is excessively low.
It's not about hitting a nerve, you are making extraordinary claims and for some reason you think other people should compile your source and disprove you instead of you showing any evidence in the first place. Why would you think that?
I think what OP was saying is that yes, it would impress _them_, and it may impress other _people_, and impressing people is generally good for business.
compared to mongodb i have no ideas how to use your database or whatever it is. with mongodb i can just download and unstall, then copy a code snippet and run it myself.
You download it and then after some time you unstall it, got it. Well, I don't think I need to worry about MongoDB then ;)
But I know what you mean. ResinDB is a library that let's you embedd a database inside of your application. It's not a service such as MongoDB. MongoDB load and keep indices in-memory. That's the fastest type of architecture you can have if you want to answer quickly to queries. To have it all in-memory.
Well, there _is_ one faster way. It's to construct a smart index file, bitmapped, stored on a SSD, where the data is laid out in such a way that reading from it is just as fast or faster than reading from an in-memory data structure. This is what Resin achieves.
i dont even know what you are selling. show me a demonstration. can i run it in nodejs ? or sell directly to those who know exacly what you are talking about and solves their particular need.
I agree that a library such as this project is not at all as consumer-friendly as an application is. Some might even call it completely unsexy. It's a component of something bigger though, something you can indeed call into from nodejs. But that's another project.
>or sell directly to those who know exacly what you are talking about
Yeah I've been thinking I should try to get a few gigs as a speaker at tech meetups or conferences to talk about this tech but I haven't yet found a good enough story to tell.
Edit: give me one more chance to describe what Resin is.
Have you heard of SQL Server LocalDB? It's proper SQL Server, but it runs inside the process of your application. It's a library that has support for SQL, fast reads and writes. It's a database like any other databasem but it's a library.
Unfortunately (or not) SQL Server LocalDB has no support for full-text search. This is why there is a marketplace for libraries such as Lucene, who make full-text search their priority. That market place has been as fixed on Lucene, an open source free software project (LGPL), as the world has been on Google for about as long a timespan. If I want to make a dent on that market I need to be as open as Lucene and as performant.
To keep up with achademia the code base of a search engine should move fast (my view). Managed code lacks the preciseness of C++ but allow you to work fast. As hell.
So, ResinDB looks very much like a much smaller (in code size) version of Lucene. We will see the coming months or so, who moves the fastest. Me or the Lucene team.
Thank you, I think I somewhat understand now. For it to be accessible for me, I'm used to something like:
installation:
npm install resin
usage:
var resin = require("resin");
var wikipedia = resin.init({file: "c:\temp\wikipedia.json", dir: "c:\resin\data\wikipedia"});
var dogs = wikipedia.query("title: dog");
// or ...
var players = resin.init({dir: "c:\resin\data\playerData"});
var oldPlayers = players.query("age > 30");
I agree to what you said whole-heartedly and I have to conlude by now that I am so not even close to being a sales person. I don't care for the psychology of a sale. I've been in many. I've only seen one or two beautiful ones.
