Current state of yacy network

[karlis-repsons]

As I saw in my yacy page, there are only 50 senior peers, is that right? I think is very little, have you been thinking about what holds people from participating more actively? Maybe summarised results?

[Low012]

Do you mean something like this? http://www.yacystats.de/ Unfortunately it is only in German so far.

I think YaCy is still not as easy to use as it should be. You maybe still need to put too much energy into it to understand it. Also the benefit is not easy to see and to communicate to people and if your index becomes really large you also need to donate quite some resources to it.

When the project was started the aim was to have a really large network of peers. Now the focus has shifted a little bit to allow people who really need it to set up their own clusters and networks. One example is Sciencenet (scroll down a little bit on http://liebel.fzk.de/).

If you have any ideas for additional use cases or ways to keep people running YaCy after the first thrill of running your own crawler is gone, please share them.

[karlis-repsons]

Well, it took me a day to set it up, go through all menus, consider the security questions, make a new Linux startup script for that and start first crawls, then look what / how happens.
I was impressed by the precision with which yacy reminded me the right links to memorised keywords and phrases of pages I crawled.

It really didn't feel good to tunnel all my traffic through yacy proxy, so I tried to set up proxy switch on firefox, but it just didn't work for some strange reason - also firefox own proxy didn't work (I had required to ask password, but firefox didn't do it). Konqueror, however worked.

I think, two things are in way of yacy deployment:
1) no outside reverse connection - who knows, with ipv6 things better, but not now with 4... Also, many networks including mine are quite slow.
2) people might feel worried about yacy caching what it should not, however, if they can't set up proxy switcher, then setting up yacy is no more easy, especially when there is the big google.

My ideas? Well... Let me ask first: is that (which one) of any considerable help, if I'd share some 50GB of data while I work at daytime and max only 70MB all of the time, if someone wrote C++ node (see below)?

C++ node:
there are circumstances when running java is not going to be accepted and, as its also my case considering another machine, I would like to set up some C software which could monitor some chosen web sites, index them and share results 24/7.

[karlis-repsons]

And there is more I didn't write down: why c++ node? Imagine this: people, who own websites could set it up to index their own www content, maintain it up-to-date (they know best what/how!) and share for all, while not wasting any resources with java vm. Well, for you to consider. I know it takes writing up completely different application (most likely, Linux daemon), but maybe it can be really useful? Comments?

[Low012]

All communication between YaCy peers works with regular HTTP messages. Even though the interface is not documented very well, there is no secret magic or hiding of anything. It should be possible to develop software to become part of the YaCy network by using different languages than Java or even use a totally different concept to collecting and storing index data. This work can't be done by the current YaCy team though since resources are pretty limited (just a few people, development in spare time).

[gox]

As I saw in my yacy page, there are only 50 senior peers, is that right? I think is very little, have you been thinking about what holds people from participating more actively? Maybe summarised results?

Noob here.

When I started to run yacy today, I wasn't expecting to see a peer count as low as 89. But rather unexpectedly, the 'freeworld' still works, albeit with none to very few results.

Within the recent climate -- where search engine giants aren't considered all benevolent entities anymore -- I got the impression that freeworld has a chance. Assuming it scales well. Does it?

- Even if it's java, there must be a lot of admins who would love to run yacy on their servers. I'll try to persuade a few. But current penetration is too low: they just haven't heard of yacy, or haven't thought of a reason to run it. However, there are lots of reasons!

- I haven't even downloaded the windows version, so this could be totally irrelevant: Yacy could be stripped down to something like the seti@home screen-saver. That would be a good marketing perspective to attract curious altruistics. Disk space is very cheap nowadays. Port forwarding could be optional. I don't think proxy cache is an issue, people would like to use yacy without it, or take the risk instead, which is little.

- I could be over-emphasizing this but yacy -- with a little bit larger freeworld -- is a viable option as open web search API. Developers wouldn't need to worry about a particular company's marketing strategies (like when Google discontinued their SOAP search API) and limitations (number of searches per day, limited api calls, etc). (I guess this is a good point to mention the LACK OF DOCUMENTATION though.)

These were the first things that came to my mind when I read your post. Surely none would be new ideas...

EDIT: Another thing tangentially related to the C++ talk: (And yes, what's up with these DHT people and their Javaphilia?) The most important issue about such a network seems to be persistency of information. I guess if data doesn't fall off the network that easily (i.e. more redundancy), freeworld would be much more usable. It should be fairly easy to implement a server with bare storage and DHT functionality (along with the search interface), with weak configurability, from scratch. Without crawling, memory footprint and network activity would be miniscule, and installation pretty easy. OTOH, while a standalone cache is possible, a standalone crawler isn't, so this might introduce additional development overhead. All in all, I think freeworld needs much more redundant storage than crawlers, as crawling is useless if the data falls off. Is this worth discussing? (Another question would be about how yacy works, which I'm totally ignorant of: Would a totally redundant node disconnecting be harmful in any way? If not, a lot of desktop users, who are not using their computers 24/7 could run caching only reduncancy nodes to the benefit of the network.)

[mick]

I am using YaCy to crawl/index/fetch results for visitors to websites I'm involved with (as one of the older posts above suggested) ..the visitors never need realize YaCy is being used (other than the prominent YaCy logo/link) and the YaCy server itself is never exposed. So far I'm still working thru the kinks with a less mission critical site.

I've seen YaCy evolve so much since I first started trying to make it useful. I'm confident it will keep heading in positive directions. Once I have my own stuff sorted out I think I'd like to try to develop some easy howtos for integrating YaCy into your website. I don't see any good guys in the field of websearch, and I think it's absurd to even think of searching the web as something that would be centralized. I think Google's model must already be looking like a dinosaur and Google is probably trying to hedge against the business of search more and more. Otherwise for all I know my thinking is... "YaCy* your our only hope." So yeah let's pray YaCy will be the next Bittorrent for WWW-wide search and DIY Google for website search one day.

I have not had a sense of what the mission or constituency of YaCy's community is. But personally I'd love to be able to say YaCy the Google Slayer one day. And I gotta admit, I say that without any humor.

I think I'm using YaCy correctly, but it still has a lot of usability issues which are more problematic than just miscommunication. The last revision update I made however was a world of difference. Time will tell if the service daemon and database are more persistent than they have been or not.

PS: It would be nice if there was a "show recent posts" link prominently displayed in this forum. I tend to not see things outside the Install/Support board.

EDITED: Woops, for some reason I thought this topic was a little fresher. Ah well, my 2 cents for the "current state of YaCy"

[cypher]

Hi Guys,

How are you making calls to Yacy from your own programming code? Does it have a web service, DLL, or other API interface? If so, any helpful hints on the docs on this?

I'm interested in doing web searches of both websites and images utilizing key words through API calls from C# code.

Obviously, I am happy to allow Yacy peers to run in my infrastructure to facilitate this as well.

Thanks!

[Sherlock]

Don't know if this helps, but the "Technology" page at yacy.net states:
"The search interface can be accessed using the OpenSearch standard" with a link to http://a9.com/-/company/opensearch.jsp

[Low012]

It is very easy to receive search results as XML (Opensearch RSS) or JSON. Do a regular websearch and then replace ".html" with ".rss" or ".json":

Examples:

As HTML:
http://4o4.dyndns.org:8080/yacysearch.html?query=example&verify=true&resource=global

As RSS:
http://4o4.dyndns.org:8080/yacysearch.rss?query=example&verify=true&resource=global

As JSON:
http://4o4.dyndns.org:8080/yacysearch.json?query=example&verify=true&resource=global