@mlarrieu thanks!

usually that sort of thing would be specified in the guide: guide.fatcat.wiki/entity_conta

but in this case it is sort of subjective and used mostly for analytics. it is inferred from other metadata fields. code is, obscurely, here: github.com/internetarchive/cho

"One site, scholar.archive.org, has PDFs going back to the 18th century. Itโ€™s empowering to look for this stuff instead of waiting for it to be socially discovered and jammed into my brain."
๐Ÿ’• @ftrain
(h/t @jstogdill for the reference)

@albertcardona @MarcusStensmyr @textfiles @giorgiogilestro whoops! we have the "editor" metadata in the catalog: fatcat.wiki/release/fyescyq5ln

but doesn't come through in search results. how would you expect this to display in search results? "(Ed.)" after the editor name? PubMed seems not to display; PLOS (publisher) shows in separate metadata box

@melissaekline @textfiles @ORCID_Org we don't have any plans to take on author profile pages ourselves, though somebody could certainly build something like that on top of the catalog API. getting human names and de-duplication correct is hard and can be harmful if done wrong.

the fatcat catalog does have a concept of "creators" (eg, authors and editors), and can be edited by anybody to update author/paper linkages. it contains a lot of ORCID-based records, but is not very complete

@Anarchy_How @textfiles it is possible to directly edit the catalog ("fatcat") by clicking through the green link below results.

if that is too much hassle, you can DM this account here on mastodon, chat in gitter.im/internetarchive/fatc, or use the contact info on the about page

@BeTongLen @textfiles it isn't perfect, but getting better all the time!

by default we only display results that we have an accessible full text copy of. you can change the "availablity" filter to "all records" to override this.

if you know of an open copy of a paper we don't have, you can click through to 'fatcat' and submit a "Save Paper Now" request for us to crawl and index it

@JigmeDatse @textfiles we are a search engine over research publications, so pretty similar to Google Scholar in a lot of ways.

we have no current intention of building high-quality author profile pages, or computing "leaderboard" style citation metrics summaries, which are features of Google Scholar.

one difference is that all of our biblio metadata and code is openly available for reuse, and we have an open search API

For reasons I can't fathom, Internet Archive Scholar got attention today, a mass of it, painting it as a "new" service. Actually, it has been out there for about a year. BUT....

If beautifully structured access to academic citations by the millions is your bag or desperately needed tool, especially ones that are ONLY left in the Wayback Machine, you are in LUCK. And this will be your favorite day. Try it.


๐‘ด ๐‘ข๐‘ฌ, ๐‘ฅ๐‘ฒ ๐‘๐‘ฑ๐‘ก ๐‘ฏ ๐‘š๐‘ป๐‘›๐‘•๐‘ฒ๐‘‘ ๐‘ฉ๐‘’๐‘ฌ๐‘ฏ๐‘‘ ๐‘œ๐‘ง๐‘‘ ๐‘ฉ ๐‘–๐‘ฌ๐‘‘๐‘ฌ๐‘‘ ๐‘ฆ๐‘ฏ ๐‘ฉ ๐‘ฎ๐‘ฐ๐‘•๐‘ฉ๐‘ฏ๐‘‘ ๐‘จ๐‘’๐‘ฉ๐‘›๐‘ง๐‘ฅ๐‘ฆ๐‘’ ๐‘ธ๐‘‘๐‘ฆ๐‘’๐‘ฉ๐‘ค ๐‘ช๐‘ฏ ๐‘ฆ๐‘™๐‘œ๐‘ค๐‘ฆ๐‘— ๐‘•๐‘๐‘ง๐‘ค๐‘ฆ๐‘™ ๐‘ฎ๐‘ฆ๐‘“๐‘น๐‘ฅ. ยท๐‘ฆ๐‘ฏ๐‘‘๐‘ผ๐‘ฏ๐‘ง๐‘‘ ๐‘ธ๐‘’๐‘ฒ๐‘ ๐‘•๐‘’๐‘ช๐‘ค๐‘ผ ๐‘ฆ๐‘Ÿ ๐‘ฉ ๐‘œ๐‘ฎ๐‘ฑ๐‘‘ ๐‘ฎ๐‘ฆ๐‘Ÿ๐‘น๐‘• ๐‘‘ ๐‘ฅ๐‘ฑ๐‘’ ๐‘ฉ๐‘๐‘ฑ๐‘ค๐‘ฉ๐‘š๐‘ฉ๐‘ค ๐‘ž ๐‘“๐‘ฎ๐‘ต๐‘‘๐‘• ๐‘ ๐‘จ๐‘’๐‘ฉ๐‘›๐‘ง๐‘ฅ๐‘ฆ๐‘’ ๐‘ฎ๐‘ฆ๐‘•๐‘ป๐‘—.

โ†’ Oh wow, my page and birdsite account get a shoutout in a recent academic article on English spelling reform. Internet Archive Scholar is a great resource to make available the fruits of academic research.



@dlakelan any features you would like to see, or content that is missing?

@naudgnit any features you would like to see, or content that is missing?

Pushed a fresh snapshot of fatcat metadata last week:

Hundreds of millions of paper, file, and journal records. More info about these dumps, and schema, at guide.fatcat.wiki/bulk_exports

python library 

Scholar contains content that we have crawled from open sources, such as OA publishers, repositories, and national libraries. Most of our Japanese metadata and content comes via the JALC DOI registrar and JSTAGE hosting site.

There is also digitized print content in archive.org, and some of that ends up in scholar. I don't know of any specific Japanese research collections there.

There are more details in "the guide": guide.fatcat.wiki/

Scholar is built on an open, editable bibliographic catalog: fatcat.wiki

Most of the records are automatically imported from our wonderful upstream sources, but any human can directly submit corrections and additions through the web interface or API. These submissions are then reviewed in the open before merging. The entire catalog is versioned and can be downloaded in bulk or synchronized using a "changelog" feed.

You can learn more about editing at:

python library 

trafilatura (github.com/adbar/trafilatura) is a nice python library that we use to extract article full text from HTML documents for indexing in scholar. It has good accuracy and recall, works with "old" HTML (eg from web archives), and pulls out metadata like title, author, and date. There are lots of similar tools, mostly focused on news articles, and trafilatura is an improvement.

Thanks to Adrien Barbaresi for maintaining it!

A less-known feature of IA Scholar is that every search result page has an RSS feed, via the link under the search bar.

Quick and easy way to keep up with a specific topic, venue, or author in your feed reader!

Scholars continue to publish papers in Latin, well in to the twenty first century! Here is a snippets of Dennis Toscano's Masters thesis from the University of Kentucky (2016), contextualizing an anonymous poem, itself in Latin, from 1741:

Opus cui titulus est "Carthago Indiarum obsessa sed non expugnata" est carmen divulgatum sine nomine auctoris saeculo duodevicesimo ad celebrandam victoriam quam Hispani a Britannis Carthagenae Indiarum anno...


Show older
Internet Archive

A Mastodon Server for Internet Archive employees and Role Accounts (Announcements)