Pipeworks | Get-Web

Get-Web

Gets content from the web. If -Tag is passed, extracts out tags from within the document. If -AsByte is passed, returns the response bytes

Getting Started
Play with Pipeworks
Connecting the Clouds
Join Windows and Web

Get-Web (View Source)

Gets content from the web, or parses web content.

Gets content from the web. If -Tag is passed, extracts out tags from within the document. If -AsByte is passed, returns the response bytes

AsMicrodata

If set, extracts Microdata out of a page

Async

If set, will request the web site asynchronously, and return the results

ContentType

The MIME content type you're requesting from the web site

Data

Any request ascii data. Data will be joined together with &, and will be sent in the request body.

Header

a hashtable of headers to send with the request.

HideProgress

If set, will not show progress for long-running operations

ItemType

If set, will get back microdata from the page that matches an itemtype

MetaData

If set, will extract all meta tags from a page

OpenGraph

If set, extracts OpenGraph information out of a page

OutputResponseHeader

If set, then a note property will be added to the result containing the response headers

ProgressIdentifier

A Progress Identifier. This is used to show progress inside of an existing layer of progress bars.

RequestBody

The Request Body. This can be either a string, or bytes

Root

The root of the website. All images, css, javascript, related links, and pages beneath this root will be downloaded into a hashtable

TextInTag

If used with -Tag, -RequireAttribute will only match tags with a given keyword in the tag

Timeout

The amount of time before a web request times out.

UseDefaultCredential

If set, will use the default user credential to connect to the web site

UseErrorAsResult

If set, the server error will be turned into a result. This is useful for servers that provide complex error information inside of XML or JSON.

If set, will use a the Net.WebRequest class to download. Otherwise, will use the xmlhttprequest. Xmlhttprequest adds some extra headers and caches GET requests, so, if you wish to avoid this, -UseWebRequest.

WebCredential

The credential used to connect to the web site

Examples

Example 1

----------------------- EXAMPLE 1 -------------------------- PS C:\># Download the Microsoft front page and extract out links Get-Web -Url http://microsoft.com/ -Tag a

Example 2

----------------------- EXAMPLE 2 -------------------------- PS C:\># Extract the rows from ConvertTo-HTML $text = Get-ChildItem | Select Name, LastWriteTime | ConvertTo-HTML | Out-String Get-Web "tr" $text

Example 3

----------------------- EXAMPLE 3 -------------------------- PS C:\># Extract all PHP elements from a directory of .php scripts Get-ChildItem -Recurse -Filter *.php | Get-Web -Tag .\?php, \?

Example 4

----------------------- EXAMPLE 4 -------------------------- PS C:\># Extract all asp tags from .asp files Get-ChildItem -Recurse | Where-Object { '.aspx', '.asp'. '.ashx' -contains $_.Extension } | Get-Web -Tag .\%

Example 5

----------------------- EXAMPLE 5 -------------------------- PS C:\># Get a list of all schemas from schema.org $schemasList = Get-Web -Url http://schema.org/docs/full.html -Tag a | Where-Object { $_.Xml.href -like '/*' } | ForEach-Object { "http://schema.org" + $_.xml.Href }

Example 6

----------------------- EXAMPLE 6 -------------------------- PS C:\># Extract out the example of a schema from schema.org $schema = 'http://schema.org/Event' Get-Web -Url $schema -Tag pre | Where-Object { $_.Xml.Class -like '*prettyprint*' } | ForEach-Object { Get-Web -Html $_.Xml.InnerText -AsMicrodata -ItemType $schema }

Example 7

----------------------- EXAMPLE 7 -------------------------- PS C:\># List the top 1000 sites on the web: Get-Web "http://www.google.com/adplanner/static/top1000/" -Tag 'a' | where-Object {$_.Tag -like "*_blank*" } | ForEach-Object { ([xml]$_.StartTag.Replace('"t', '" t')).a.href }

Provided By

Powered With