2014-11-04 71 views
2

我想從內部推文捕獲圖像的URL。豬捕獲與正則表達式匹配的字符串

REGISTER 'hdfs:///user/cloudera/elephant-bird-pig-4.1.jar'; 
REGISTER 'hdfs:///user/cloudera/elephant-bird-core-4.1.jar'; 
REGISTER 'hdfs:///user/cloudera/elephant-bird-hadoop-compat-4.1.jar'; 

--Load Json 

loadJson = LOAD '/user/cloudera/tweetwall' USING com.twitter.elephantbird.pig.load.JsonLoader('-nestedLoad') AS (json:map []); 
B = FOREACH loadJson GENERATE flatten(json#'tweets') as (m:map[]); 
tweetText = FOREACH B GENERATE FLATTEN(m#'text') as (str:chararray); 

日期中間看起來是這樣的:

(@somenameontwitter your nan makes me laugh with some of the things she comes out with like http://somepics.com/my.jpg) 

然後我嘗試做以下,僅保留圖像的URL後面:

x = foreach tweetText generate REGEX_EXTRACT_ALL(str, '((http)(.*)(.jpg|.bmp|.png))'); 

dump x; 

,但似乎並沒有工作。我也嘗試過濾器無濟於事。

試圖與上述即使*返回空結果()或(())

我不擅長用正則表達式和漂亮的新豬所以它可能是我想的東西簡單在這裏,我只是沒有看到。

更新

例如輸入數據

{"tweets":[{"created_at":"Sat Nov 01 23:15:45 +0000 2014","id":5286804225,"id_str":"5286864225","text":"@Beace_ your nan makes me laugh with some of the things she comes out with blabla http://t.co/b7hjMWNg is an url, but not a valid one http://www.something.com/this.jpg should be a valid url","source":"\u003ca href=\"http:\/\/twitter.com\/download\/iphone\" rel=\"nofollow\"\u003eTwitter for iPhone\u003c\/a\u003e","truncated":false,"in_reply_to_status_id":52812992878592,"in_reply_to_status_id_str":"522","in_reply_to_user_id":398098,"in_reply_to_user_id_str":"3","in_reply_to_screen_name":"Be_","user":{"id":425,"id_str":"42433395","name":"SAINS","screen_name":"sa3","location":"Lincoln","profile_location":null,"description":"","url":null,"entities":{"description":{"urls":[]}},"protected":false,"followers_count":92,"friends_count":526,"listed_count":0,"created_at":"Mon May 25 16:18:05 +0000 2009","favourites_count":6,"utc_offset":0,"time_zone":"London","geo_enabled":true,"verified":false,"statuses_count":19,"lang":"en","contributors_enabled":false,"is_translator":false,"is_translation_enabled":false,"profile_background_color":"EDECE9","profile_background_image_url":"http:\/\/abs.twimg.com\/images\/themes\/theme3\/bg.gif","profile_background_image_url_https":"https:\/\/abs.twimg.com\/images\/themes\/theme3\/bg.gif","profile_background_tile":false,"profile_image_url":"http:\/\/pbs.twimg.com\/profile_images\/52016\/DGDCj67z_normal.jpeg","profile_image_url_https":"https:\/\/pbs.twimg.com\/profile_images\/526\/DGDCj67z_normal.jpeg","profile_banner_url":"https:\/\/pbs.twimg.com\/profile_banners\/424395\/13743515","profile_link_color":"088253","profile_sidebar_border_color":"D3D2CF","profile_sidebar_fill_color":"E3E2DE","profile_text_color":"634047","profile_use_background_image":true,"default_profile":false,"default_profile_image":false,"following":false,"follow_request_sent":false,"notifications":false},"geo":null,"coordinates":null,"place":null,"contributors":null,"retweet_count":0,"favorite_count":1,"entities":{"hashtags":[],"symbols":[],"user_mentions":[{"screen_name":"e_","name":"\u2601\ufe0f effy","id":3998,"id_str":"398","indices":[0,15]}],"urls":[]},"favorited":false,"retweeted":false,"lang":"en"}]} 
+0

你想從每個推文中提取圖像的網址嗎?即最終輸出應該是「http://somepics.com/my.jpg」? – 2014-11-04 13:28:01

+0

正確,這就是我想 – Havnar 2014-11-04 13:28:53

回答

0

我設法得到它的工作(雖然我懷疑它是完全最佳)

x = foreach tweetText generate REGEX_EXTRACT(str,'(http://.*(.jpg|.bmp|.png))',1) as image; 


filtered = FILTER x BY $0 is not null; 


dump filtered; 

所以最初的問題只是正則表達式(以及我對這個主題缺乏瞭解)。

感謝您的協助sivasakthi jayaraman

1

試試這個,讓我知道,如果這個工程

x = foreach tweetText generate REGEX_EXTRACT(str,'.*(http://.*.[jpg|bmp|png])',1); 
DUMP x; 
+0

,這給了我一些更好的結果,但它還沒有。 這是我找回了一些示例:'() (http://t.co/Vbiq6ZuvzB)RT進入ENTRAN) () () (http://t.co/TQu8XGg) (http://t.co/EYp) () (http://t.co/g0p) () (http://t.co/efo13URHBg HTTP) () (HTTP:/ /t.co/DJU5KlsiCr HTTP) () () (http://t.co/HUVPF9j) (http://t.co/iipidujn) () () (HTTP:// t.co/Xd6NqApcnC http) () () () () () (http://t.co/tXQT891XA5 HTTP) () () () (http://t.co/b7hjMWNg)' – Havnar 2014-11-04 13:35:22

+0

你能粘貼您的JSON文件。我想檢查圖像url的所有輸入? – 2014-11-04 13:42:40

+0

在問題 – Havnar 2014-11-04 13:48:03