DeepWeb Ad

Monday, August 31, 2015

Transportation, History and CyberWar Search Engines

Transportation

FAA Flight Delay Information — A map of the United States with flight delay information from the nation’s largest airports.
NTSB Accident Database and Synopses — The National Transportation Safety Board’s database of aviation accidents, ranging from 1962 to present.
NTSB Aviation Accident Database— Aviation accident data from the National Transportation Safety Board
National Highway Traffic Safety Administration — A database of car and car part defects, searchable by item number or car make and model.
SaferCar.gov — Crash test safety ratings for automobiles since 1990.
flightwise.com — Real-time flight tracking, with support for Google Earth.
FlightAware — Fee flight tracking with history, graphs, and maps.

History

Ohio State – Compiles lots of Civil War info on troop movements, camps, battles, etc.

Cyber War / InfoSec

Iron Geek irongeek.com — An excellent library of videos explaining many facets of InfoSec and hacking & security
Security Tube – securitytube.net — A large library of videos covering many topics in InfoSec, cyberwar, and most of the hacking conferences.
DefCon — The main hackers Con, so well known that now the Feds send their folks here and it has become a wild west training ground for coming trends. Archives go back to Defcon 1. They are now on Defcon 20, I think.
semanticommunity.info– General military hardware overview. Some deeper links to deep web
Shodan – The deep web search for what things are connected to the internet.  Controversial, but a good tool.

Hey people. If you know of a useful deep web resource, put a comment below and share the love!

Science and Academic search engines

Science and Academic

(Scholarly Paper search engines now have their own page)
ScienceResearch.com — Searchable access to scientific journals and databases.
Academic Index – Main search is a filtered Google search aimed at high authority rank sites, mainly .edu and .gov which filters a great deal out. Second search ties into deep web academic and non-academic databases skewed to librarians and educators.
Science.gov — Gateway to  science info provided by US government agencies.
VideoLectures.net –  Phenomenal video lecture coverage from high authority rank sources.  A great go-to place to find peer-reviewed, conference presented, in depth coverage of a topic at a conference. A nice bonus, is the presentation slides are shown separately, and you can jump to slides of interest to you. Heavily technology based, and 66% is in English. Most lectures 45 minutes or longer.
WebCASPAR — A horrible interface to an alleged wealth of statistical info on science and engineering. I found the site slow, cludgy and designed around 1965 run off of candle power.  From their website:”The WebCASPAR database provides easy access to a large body of statistical data resources for science and engineering (S&E) at U.S. academic institutions. WebCASPAR emphasizes S&E, but its data resources also provide information on non-S&E fields and higher education in general. ”
The Complete Work of Charles Darwin — Charles Darwin’s published works,  search-able and available online. He’s still old and his works still ramble. Scanning didn’t help him much.
USGS Real-Time Water Data — Real-time map of streamflow and water quality data of the USA’s rivers and reservoirs.
USGS Earthquake Hazards Program — Showing real-time earthquake data. Focus on US but covers world as well.
IEEE Publications (Commercial) — Contains over 1.4 million documents from the Institute of Electronics and Electrical Engineers.
Society of Petroleum Engineers Archive (Commercial) — Petroleum engineers technical papers.
Arxivarxiv.org/ — Cornell University repository. Access to 700,000+  technical papers on everything from quantitative biology to computer science. Appears to offer full text in several formats.
VADLO – www.vadlo.com/ — Life Science Search Engine. Very hit and miss. Don’t have high expectations.
Deep Dyve  (Commercial) www.deepdyve.com   DeepDyve has aggregated millions of articles across thousands of journals from the world’s leading publishers, including Springer, Nature Publishing Group, Wiley-Blackwell and more. Haven’t paid the premium to give it a test ride, if someone has, please write a review below.
Data Mining Data sources – http://www.kdnuggets.com/datasets/index.html  – Links to gobs of free and commercial datasets used for data mining.

Medical and Health search engines

Medical and Health

PubMed  — The U.S. National Library of Medicine contains over 16 million citations from MEDLINE and other life science journals going back to the 1950s. Contains links to full-text articles and external resources. Supposed to be the best damn resource for medical out there.
National Institute for Health Research Archive — http://www.nihr.ac.uk/Pages/NIHRArchive.aspx Database of ongoing or completed projects funded by the British  NHS.
National Institutes of Health — Encyclopedia of health topics. More of a kindof-deep-web resource, as Google has this indexed.
American Hospital Directory — Index of US hospital information.
Globalhealthfacts.org — Searchable world health information, by country, disease, condition, program, or demographic. Quickly lay out the conditions in a country.
Centers for Disease Control and Prevention Data and Statistics — Statistical health information according to the CDC. Not true deep web, but useful.
New England Journal of Medicine  (commercial) — A Leading medical journal with full text past issues available online. Commericial, but you can access most for free.

Law and Politics search engines

Law and Politics

THOMAS (Library of Congress) — Legislative information from the Library of Congress.
Law Library of Congress — Allegedly, the largest collection of legal materials in the world, over 2 million volumes.
Global Legal Information Network — Laws, regulations, judicial decisions, and other legal sources.
FindLaw — Free legal database, with collections of cases and codes, legal news.
Office of Postsecondary Education Security Statistics — Contains college campus crime statistics, sortable.
Bureau of Justice Statistics — http://bjs.ojp.usdoj.gov/ Legal and judicial statistics, everything from crime to law enforcement.
The Avalon Project at Yale Law School — Documents in law, history, and diplomacy.
US Congressional Documents and Debates, 1774–1873 (Library of Congress) — The historical Congressional records, bills, documents, statutes, journals, and debates from LOC.
Lobbying Database — Who spent what on firms who have spent lobbying money from 1998. The US, with the finest Congress money can buy!
Legislative Activities — Synopsis of House of Representatives’ bill summary, status, public laws, and votes.
heinonline.org  – (commercial) – Claims to be the ‘worlds largest image based database of legal documents’ . I was able to find an obtuse document on using Bayes Theorem for fact finding in a criminal case.
Project Vote Smart — Government officials and election candidates database, order by last name or ZIP code.

International Search Engines

International Search Engines

International Data Base (IDB) — Statistical tables and demographic information for countries and areas.
FIRST http://first.sipri.org/ Military aggression database and weapons holdings.
Economics of Tobacco Control — Information regarding tobacco usage and policy for180 countries.
Country Indicators for Foreign Policy —Statistical tables on countries’ foreign policies.
World Bank Data — Key development data and statistics for countries and worldwide groups.
CIA Factbook — Reference materials containing information on every country in the world.
US International Trade Statistics — International trade statistics,by country or type of good.
US Foreign Trade Highlights — Information of US international trade.
Energy Information Administration International Energy Data and Analysis — Energy balances sorted by country. Explains why the US gives huge amounts of cash to oil producing countries.

Finance and Government Search Engines

Bankrate.com — Database of interest rates for different loan types, mortgages, and savings accounts.
InvestIQ — Market data from around the world in regions.
BigCharts — Quotes and performance charts on different stocks and mutual funds.
SmartMoney.com Tools — A portal of for stock analysis tools.
NASDAQ Trader — A database of stock data from the NASDAQ stock exchange.
SEC Info — EDGAR and SEC filings searchable by name, industry, SIC code, etc.
EDGAR Online — SEC filings searchable by ticker or company name.

Government Search Engines

Copyright Records (LOCIS) — Online copyright records, documents, serials, and multimedia.
American FactFinder — Aggregate census bureau data to be searched by city, county, or ZIP code.
FedStats — Gateway for statistics on 100 US federal agencies.
United States Patent and Trademark Office — Database of patents,  full-text and full-page images.
Historical Census Browser — Repository of historical US census data. Going back to 1790 compiled by the University of Virginia.
Geospatial One Stop — Awesome GIS data warehouse of geographic data, shapefiles, imagery, and displayable on maps.
Grants.gov — Grant opportunities, from everything under the sun.
Technology Opportunities Program Grants Database — Listing of technology grants, peruse by keyword, state, and year.
United States Government Printing Office (GPO) — I mentioned this earlier, they seem to have everything. A search engine for mutliple government databases: US budgets, campaign reform hearings, code of federal regulations, congressional bills, etc
CIA Electronic Reading Room — The usual uninteresting declassified CIA documents.
POW/MIA Databases and Documents — Info on POWs and MIAs.
ZIP+4 Lookup — US ZIP codes and ZIP+4 codes

Economic and Job Search Engines

EUROPA Press Release Database — Database of press releases by the European Union.
FreeLunch.com —Directory of free economic data.
Bureau of Labor Statistics — Job-based and consumer economic info from the US Department of Labor.
Salary Wizard Calculator — Tools that shows national average salaries adjusted by location for different jobs.
Economagic — A data directory containing over 200,000 econ files.
Penn World Tables — National income data for all countries for the years 1950-2007.
America’s Job Bank — Database of jobs and resumes.
USAJOBS — Portal of data on federal government jobs.
Regional Economic Conditions (RECON) — Economic data available by state, county, and MSA.

Deep Web Search engines 2015

Books Online

Archive.org – Has books online in epub, txt, and pdf formats. The collection encompasses others such as Gutenberg Press, etc. So this is the best site to start with. Again, this makes my top 10 websites. Share the love.
Hathi Trust – http://www.hathitrust.org/  — a partnership of major research institutions and libraries working to ensure that the cultural record is preserved and accessible long into the future. There are more than sixty partners in HathiTrust, and membership is open to institutions worldwide.
Books.google.com – They are putting the squeeze on all the book scanning businesses. They want to scan the world to add it to the Google Borg. You will be assimilated.
The Online Books Page — A searchable database of more than 28,000 English works with full text available for free online.
Bibliomania — A database of free literature from more than 2,000 classic texts. Archive.org crushes this.
Project Gutenberg — The granddaddy of online books with a catalog of more than 20,000 free books with full text available online. Included in Archive.org.
The National Academies Press — Only about 3,000 free books online and ~900 for-sale PDFs.
ebrary — A database of about 20,000 full-text books. Focusing on academia and business.
UNZ – An odd collection of periodicals and book scanned.
Get Abstracts (commercial) — Large online library of more than 8,000 business book summaries. It is the most efficient way to get the best business titles.
Getty Research Institute – http://www.getty.edu/research/library/ – The Getty Research Institute library collections include over one million books, periodicals, study photographs, and auction catalogs as well as extensive special collections of rare and unique materials. Focusing on art history, architecture, and related fields, they begin with the archaeology of prehistory and extend to the contemporary moment.

Newspaper Archives Online

US newspaper coverage – ResearchGuides lists links to many newspapers
Library of Congress Newspaper Resource List – LOC does a great job getting the list together of wonderful newspaper archives.
NewspaperArchive.com(commercial) – Known for a large collection. I haven’t used it, so I can’t confirm this.
xooxleanswers.com – Great list of newspaper archives from Xooxle. Good list and a funky name. Two thumbs up.
University of Penn Newspaper Archive – List of US newspaper archives and dates. Looked like a deeper list of Texas newspapers, so this effort may be a deep comprehensive list.
Australia Newspaper archives – http://trove.nla.gov.au/ndp/del/about – A phenomenal newspaper archive of most Aussie newspapers from 1820’s going to 1980’s. Including Taz as well.

Audio Books Online

LibreVox.org – Huge selection of audio recordings and AudioBooks read by volunteer voice artists

Videos

www.liveleak.com – A Video news aggregator of citizen supplied videos. Great for OSInt in foreign countries. The site appears to have some good aggregation functions to turn randomly submitted videos into a logical collection around a topic.  Proven useful for aggregating the Feb 15, 2013 meteorite strike in Russia.
VideoLectures.net – High quality videos of science based lectures.

Business Deep Web Engines

AAAAgencySearch.com — Advertising agencies via the American Association of Advertising Agencies.
Agency ComPile — Advertising and marketing agencies.
Alibaba – An international marketplace of businesses looking for businesses.
Kompass — Business to business search engine.
GPO Access Economic Indicators — Gateway for economic indicators from January 1998 to present.
Government Printing Office — Big catalog of stuff published by the Government Printing office. Has business stuff but much much more. Environmental reports, legal docs, nature stuff. Hell, I typed in ‘mushroom’ and pulled up 34 entries.
Hoover’s — The Big Boy of info on businesses.
ThomasNet — Just an industrial product search directory.
SBA Loan Data — Loan program approval activity from the Small Business Administration.
GuideStar.org — A searchable database of non-profit organizations including 501c.
Slideshare.net – A searchable database for slides uploaded by crowdsourcing.

Consumer Engines

US Consumer Products Safety Commission Recalled Products — Listing of products, sortable by company name.
Melissa Data — Comprehensive directory of demographic data, sortable by ZIP code.
Kelley Blue Book — A  guide to pricing new and used vehicles.
Edmunds — A recognized and established guide to pricing new and used vehicles.
Consumer Reports — (commercial)  A trusted guide to product reviews, including autos, appliances, electronics, computers, personal finance, etc. I use it.

Find People & Background Checks

Pipl.com– for finding people
zabasearch.com – finding people
Intellus (commercial) – Finding people plus background checks on people and other features.
US Search (commercial) – Finding people plus background checks on people.
123 People (commercial) – A multi search engine built around finding people.
Integrascan – Finding people plus background checks on people.
State of Texas DOT Criminal Background Check – The central background check for felonies provided by the state. Most misdemeanours don’t show up.
socialcatfish.com – Has background check capability plus a photo reverse lookup. Deeper level of search for subscription.

Speciality Deep Web Engines

Infomine – comprehensive virtual library and reference tool for academics. But I’ve found it not too useful, better engines out there.
Archive.org – Huge behemoth of media now public domain – rare books, sound recordings, video, 20 year archived images of all old websites, and free audio books! Makes my top 10 list. (and my top 3)
WWW virtual library – a listing of indexes to industries. Need to know about Architecture? Biochemical war? Zoology? This may get you there.
FindArticles.com – FindArticles has articles from about 500 periodicals with coverage back to 1998, and is completely free of charge.
Library of Congress – loc.gov  – Phenomenal digitized archives, “American Memory” especially interesting. Includes a good newspaper archive.
National Security Archive – Declassified papers and such. In their words – “National Security Archive Electronic Briefing Books provide online access to critical declassified records on issues including U.S. national security, foreign policy, diplomatic and military history, intelligence policy, and more.  Updated frequently, the Electronic Briefing Books represent just a small sample of the documents in our published and unpublished collections.
www.osti.gov – Government research archives, if your tax dollars paid for it, the results are here. Also a huge collection of science presentation videos.
US Geologic Survey – Imagery and Maps galore. 3 portals to fetch data, EarthExplore, Glovis and Seamless. Extremely complex.  There are tutorials on how to get free aerial photos over at learninggis.com.
US National Map by USGS – The source for current geospatial data from the USGS.
http://adswww.harvard.edu/ – Physics and Astronomy data engine for academic papers
http://databank.worldbank.org/data/home.aspx  – Specialty statistical data on all kinds of subjects, from countries GDP to levels of blindness.
http://www.quandl.com/ – An awesome collection of 9,000,000 of financial, economic, and social datasets.

Cluster Analysis Engine

TouchGraph – A brilliant clustering tool that shows you relationships in your search results using a damn spiffy visualization. The smart way to use it, is to let it help you find new sources to your search topic. I have to add, the wiggly effect on the visualization is damn cool, just grab the center item and move it to understand what I’m talking about. (sometimes it doesn’t wiggle, however. Java issue?)
Yippy.com – A useful, non-graphical clustering of results. Give it 2 minutes of your time to understand how it works and it will give back hours of saved research time.
www.quintura.com – (appears down 12JUN15) An interesting data mining search engine that shows related words to your target. Good if you aren’t sure what the exact name of the subject is you seek
www.navagent.com/ – Not a web based search engine, requires you download software. Highly rated, very interesting especially to the 35F intel types.

Multi Search engines

Deeperweb.com – This is my favorite search engine. It breaks your results down into categories – general web, blogs, news, academic, cloud, metrics, research, etc. This allows you to quickly focus on the type of answer you were looking for.  Makes my top 10 websites!
Zuula.com – nice multi engine aggregator
Surfwax – They have a 2011 interface for rss and a 2009 interface I think is better. Takes 60 seconds to understand how to use it.
Dogpile – another multi engine aggregator
Scout Project- scout.wisc.edu — Since 1994, the Scout Project has focused on developing better tools and services for finding, filtering, and presenting online information and metadata.
www.findsmarter.com – You can filter the search by domain extension, or by topic which is quite neat. Sources Yahoo, Bing, Wiki, Blekko, and Alltheweb

Deep Web Search Engines

Where to start a deep web search is easy. You hit Google.com and when you brick wall it, you go to scholar.google.com which is the academic database of Google.
After you brick wall there, your true deep web search begins. You need to know something about your topic in order to choose the next tool. To be fair, some of these sites have improved their index-ability with Google and are now technically no longer Deep Web, rather kind-of-deep-web. However, there are only a few that have done so.  I recommend you use your browsers ‘search’ option to locate on this page your topic of interest, as the page has gotten long.
To all the 35F and 35G’s out there at Fort Huachuca and elsewhere, you will find some useful links here to hone in on your AO.

If you find a bad link, Comment the link below. I try to keep this up to date, but only with your help!

Last updated August 12, 2015

Deep problems in the deep web

When the Silk Road marketplace was shut down in October 2013, many felt the arrest of its owner Ross Ulbricht signaled the end of the dark market as means to sell illegal and unauthorised goods including counterfeits. The intervening months have shown this to be demonstrably not the case, with many more competing markets being launched, including a new and improved Silk Road 2.0.

Anonymous market place in the deep web
Dark market sites are not part of the regular, common-or garden-internet which most of us inhabit on a day to day basis as they can only be accessed through the Tor network.
The Tor anonymising network is a system of pseudo-anonymous networking, originally sponsored by the U.S. Naval Research Laboratory that allows users to access various internet resources such as web sites and forums with some degree of privacy. Tor also enables access to a number of hidden resources inaccessible to normal internet users. Commonly called “onion sites”1,
these locations feature content often excluded from the ‘clear net’.
Hidden onion sites on Tor are generally only accessible to those who have installed the Tor browser bundle2, a specially designed set of software which allows access to Tor through a modified version of Firefox. However, regular web users can take advantage of services such as tor2web.org and onion.to to access onion sites. These services act as a web-based proxy gateway to Tor and allow access to sites such as The Tor Directory and the Hidden Wiki3 (directories of many of the most popular services hosted via Tor) via a slightly amended URL4
However, it is vital to note that while using such methods provides simple access to onion sites, it does not provide any of the anonymity and protections that the full use of the Tor browser affords users. Using a service such as tor2web also places trust in the operator of the proxy. As such, any visits to the Tor network using a web-based proxy should be employed with caution (and indeed, a recent exploit of the Tor browser itself shows that anonymity cannot be guaranteed when using the network ).5
Browsing these sites can be something of a shock for the average web user, whose idea of living dangerously on the internet probably amounts to checking their Facebook feed while in a meeting with their boss. The world of Tor is a far darker place, with hit men offering to take out your unfaithful spouse, alongside drug dealers and pornographers. Throw in a healthy dash of scammers and hackers and five minutes with the Tor browser will convince you that you are not in Kansas anymore.
The primary means of carrying out transactions in the Tor world is through the transfer of Bitcoin, a decentralised crypto-currency that adds another layer of anonymity to proceedings.
It is perhaps unsurprising that such an environment is also an attractive potential market for counterfeiters. Using dark markets, such as Silk Road 2.0 or Hydra marketplace, sellers are able to advertise and sell a swathe of counterfeit products, with few of the risks associated with operating on the clear net.

Available in the deep web

A quick sweep of a few of these sites, turned up listings for openly counterfeit designer clothes, bags, sunglasses and watches, offered for a fraction of the price of the real item.
Watches and jewellery are readily available

Cigarettes and prescription style medication such as Viagra are also widely available on many dark market sites. These may come from grey market channels, or could be totally counterfeit and harmful to health.
Counterfeit drugs are a big problem

Other products available include counterfeit coupons offering huge discounts at a number of web and bricks and mortar stores, as well as – presumably stolen – memberships to online services such as Netflix, Hulu, or Xfinity Online.

While the quality of these products found was not tested, many users seemed happy with what’s on offer, providing feedback in the same way that a user of Ebay or Amazon might. Sellers are then able to build up trust and reputation on a site, a very important attribute when sellers are effectively untraceable and buying anything involves a significant risk of completely losing every penny spent should the seller disappear.
To manage this risk, many sites have implemented sophisticated systems of escrow, that do not release payments until the buyer is happy with the received product. Such systems have been very popular since they were developed and have significantly reduced the risk of buying materials from dark market sites.
For rights holders, the presence of counterfeit versions of their products pose obvious reputational risks, especially when featured alongside listings of illegal drugs or weapons that make up much of the rest of these dark market sites.
At present, the clear net still offers sufficient avenues for counterfeiters to sell their product on sites such as Aliexpress, DHgate and iOffer. However as enforcement actions against such sites become more common place, the unregulated dark markets will offer a potential refuge to those engaged in the sale of counterfeit goods.
The anonymous nature of the Tor network and Bitcoin transactions makes it far more difficult to shut down sites or investigate vendors than on the wider, public internet.
However, when infringing products are found on Tor, there is scope for a carefully designed system of test purchases to provide additional information and data on those offering  products through these illegitimate channels.

Deep Web Movie

Deep Web is a 2015 documentary film directed by Alex Winter, chronicling events surrounding Silk Road, Bitcoin and politics of the Dark Web.
Covering the trial of Ross Ulbricht, the documentary features interviews with Wired writer Andy Greenberg and developer Amir Taaki. Winter is best known on screen for playing Bill from the Bill & Ted film series; Deep Web features narration from his former co-star and Bitcoin enthusiast Keanu Reeves.  the film premiered at the 2015 South By Southwest film festival,[2] and aired on May 31, 2015, on the Epix network

Dig Into the Deep Web

Mention the “Deep Web” and most people will instantly associate it with the part of the Internet used for nefarious and illegal activities. For others, it is this inaccessible side of the Web, the one that requires a lot of technical skill and know-how to reach. Although these assumptions are somewhat correct, they only cover a small portion of the Deep Web as a whole.

Two sides of the coin
Anonymity is the main feature of the Deep Web, and there are plenty of people who would want to use and abuse that. For example, people who want to shield their communications from government surveillance may want to take refuge in darknets. Whistleblowers, like Edward Snowden, can share vast amounts of insider information to journalists without leaving a paper trail. Dissidents in restrictive regimes may need anonymity in order to safely let the world know what’s happening in their country.
On the flipside, those with malicious intentions can also greatly benefit from this anonymity. For example, drug sellers wouldn’t want to set up shop in an online location where law enforcement can easily determine their IP address. The same could be said for those engaged in other illegal activities like selling contraband and stolen goods.
Digging into the Deep Web
We decided to look further down the rabbit hole to get more information about the illegal activities and services offered in the Deep Web. To get information, we employed our system, called the Deep Web Analyzer (DeWa). DeWa is responsible for collecting URLs linked to the Deep Web, including TOR- and I2P-hidden sites and Freenet resource identifiers, and trying to extract relevant information tied to them like page content, links, email addresses, HTTP headers, and so on.
So far, we’ve collected more than 38 million events that account for 576,000 URLs, 244,000 of which bear actual HTML content.
DeWa also has a feature that alerts us if hidden services get a lot of traffic or if there is a large hike in number of sites. This is especially helpful in finding new malware families of cybercriminals who use TOR-hidden services to hide the more permanent parts of their infrastructures.
Cybercrime in the Deep Web
Among our observations was the fact that light drugs (read: cannabis) were the most-exchanged goods, followed by pharmaceutical products like Ritalin and Xanax, hard drugs, and even pirated games and online accounts.


The Deep Web is also home to Bitcoin and money-laundering services. Bitcoin offers a level of anonymity for users. As long as they don’t link their wallet code to their real identities, they are, to some extent, anonymous. Nonetheless, Bitcoin transactions are public, which means investigators can still examine them. Numerous services have sprouted in the Deep Web, offering to move Bitcoins through a network via micro transactions. Paying a handling fee will result in the customer getting the same amount of money but with the added bonus of having transactions that are harder to track or pin down.

The challenge of the Deep Web
Anonymity in the Deep Web will continue to raise a lot of issues and be a point of interest for both law enforcers and Internet users who want to circumvent government surveillance and intervention. Right now, there seems to be a race between “extreme libertarians” and law enforcement agencies, with the former trying to find new ways to become even more anonymous and untraceable.