Xjyuk

Tikzpicture doesn't display correctly

Do 3/8 (37.5%) of Quadratics Have No x-Intercepts?

What is a good example for artistic ND filter applications?

How did the SysRq key get onto modern keyboards if it's rarely used?

Why does the Eurostar not show youth pricing?

Convert graph format for Mathematica graph functions

Is there a word to describe someone who is, or the state of being, content with hanging around others without interacting with them?

Is it safe if the neutral lead is exposed and disconnected?

Newton's cradles

Piece of chess engine, which accomplishes move generation

What clothes would flying-people wear?

Why were contact sensors put on three of the Lunar Module's four legs? Did they ever bend and stick out sideways?

Should I accept an invitation to give a talk from someone who might review my proposal?

Blank spaces in a font

How to efficiently shred a lot of cabbage?

Would people understand me speaking German all over Europe?

Did Vladimir Lenin have a cat?

Alternatives to minimizing loss in regression

Narset, Parter of Veils interaction with Aria of Flame

Semen retention is a important thing in Martial arts?

What is the meaning of "stationarity of statistics" and "locality of pixel dependencies"?

Unknown indication below upper stave

How do I make my photos have more impact?

8086 stack segment and avoiding overflow in interrupts

What is the meaning of “stationarity of statistics” and “locality of pixel dependencies”?

Where can I find the proof of the universal approximation theorem?Is the traditional meaning of “strong AI” outmoded?What is the difference between a Convolutional Neural Network and a regular Neural Network?What is the difference between a receptive field and a feature map?What is a bad local minimum in machine learning?Pixel-Level Detection of Each Object of the Same Class In an ImageWhat is the difference between latent and embedding spaces?How to debug and find neurons that most influenced a pixel in the output image?What is “dense” in DensePose?Understanding the reconstruction loss in the paper “Anomaly Detection using Deep Learning based Image Completion”What is the meaning of the words 'bias' and 'variance' in RL?

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;

4

$begingroup$

I'm reading the ImageNet Classification with Deep Convolutional Neural Networks paper by Krizhevsky et al, and came across these lines in the Intro paragraph:

Their (convolutional neural networks') capacity can be controlled by varying their depth and breadth, and they also make strong and mostly correct assumptions about the nature of images (namely, stationarity of statistics and locality of pixel dependencies). Thus, compared to standard feedforward neural networks with similarly-sized layers, CNNs have much fewer connections and parameters and so they are easier to train, while their theoretically-best performance is likely to be only slightly worse.

What's meant by "stationarity of statistics" and "locality of pixel dependencies"? Also, what's the basis of saying that CNN's theoretically best performance is only slightly worse than that of feedforward NN?

convolutional-neural-networks terminology papers

share|improve this question

edited 3 hours ago

nbro

5,72744 gold badges1515 silver badges3232 bronze badges

asked 8 hours ago

Shirish KulhariShirish Kulhari

12111 bronze badge

New contributor

Shirish Kulhari is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

add a comment | 

4

$begingroup$

I'm reading the ImageNet Classification with Deep Convolutional Neural Networks paper by Krizhevsky et al, and came across these lines in the Intro paragraph:

Their (convolutional neural networks') capacity can be controlled by varying their depth and breadth, and they also make strong and mostly correct assumptions about the nature of images (namely, stationarity of statistics and locality of pixel dependencies). Thus, compared to standard feedforward neural networks with similarly-sized layers, CNNs have much fewer connections and parameters and so they are easier to train, while their theoretically-best performance is likely to be only slightly worse.

What's meant by "stationarity of statistics" and "locality of pixel dependencies"? Also, what's the basis of saying that CNN's theoretically best performance is only slightly worse than that of feedforward NN?

convolutional-neural-networks terminology papers

share|improve this question

edited 3 hours ago

nbro

5,72744 gold badges1515 silver badges3232 bronze badges

asked 8 hours ago

Shirish KulhariShirish Kulhari

12111 bronze badge

New contributor

Shirish Kulhari is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

add a comment | 

$begingroup$

I'm reading the ImageNet Classification with Deep Convolutional Neural Networks paper by Krizhevsky et al, and came across these lines in the Intro paragraph:

Their (convolutional neural networks') capacity can be controlled by varying their depth and breadth, and they also make strong and mostly correct assumptions about the nature of images (namely, stationarity of statistics and locality of pixel dependencies). Thus, compared to standard feedforward neural networks with similarly-sized layers, CNNs have much fewer connections and parameters and so they are easier to train, while their theoretically-best performance is likely to be only slightly worse.

What's meant by "stationarity of statistics" and "locality of pixel dependencies"? Also, what's the basis of saying that CNN's theoretically best performance is only slightly worse than that of feedforward NN?

convolutional-neural-networks terminology papers

share|improve this question

edited 3 hours ago

nbro

5,72744 gold badges1515 silver badges3232 bronze badges

asked 8 hours ago

Shirish KulhariShirish Kulhari

12111 bronze badge

New contributor

Shirish Kulhari is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

share|improve this question

edited 3 hours ago

nbro

5,72744 gold badges1515 silver badges3232 bronze badges

asked 8 hours ago

Shirish KulhariShirish Kulhari

12111 bronze badge

New contributor

Shirish Kulhari is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

share|improve this question

edited 3 hours ago

nbro

5,72744 gold badges1515 silver badges3232 bronze badges

asked 8 hours ago

Shirish KulhariShirish Kulhari

12111 bronze badge

New contributor

Shirish Kulhari is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

edited 3 hours ago

nbro

5,72744 gold badges1515 silver badges3232 bronze badges

edited 3 hours ago

nbro

5,72744 gold badges1515 silver badges3232 bronze badges

asked 8 hours ago

Shirish KulhariShirish Kulhari

12111 bronze badge

New contributor

Shirish Kulhari is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

asked 8 hours ago

Shirish KulhariShirish Kulhari

12111 bronze badge

1 Answer
1

active

oldest

votes

2

$begingroup$

locality of pixel dependencies probably means that neighboring pixels tend to be correlated, while faraway pixels are usually not correlated. This assumption is usually made in several image processing techniques (e.g. filters). Of course, the size and the shape of the neighborhood could vary, depending on the region of the image (or whatever), but, in practice, it is usually chosen to be fixed and rectangular (or squared).

stationarity of statistics probably means that the values of the pixels do not change over time.

With while their theoretically-best performance is likely to be only slightly worse the authors probably thought that, theoretically, CNNs are not as powerful as feedforward neural networks. However, both CNNs and FFNNs are universal function approximators (but, at the time, nobody probably had yet investigated seriously the theoretical powerfulness of CNNs).

share|improve this answer

answered 4 hours ago

nbronbro

5,72744 gold badges1515 silver badges3232 bronze badges

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I really think in this case stationarity isnt regarding time, but spatial location (given no time exists-- im assuming you meant over the dataset) I think theyre saying alot of local statistics are common across multiple areas of the image (therefore a single convolved filter can be a useful featurization of multiple fields rather than just one)
  $endgroup$
  – mshlis
  1 hour ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  @mshlis You might be right. This was my interpretation, given the usual meaning of stationarity (e.g. in reinforcement learning). However, if your interpretation is correct, then their terminology is highly confusing or misleading.
  $endgroup$
  – nbro
  59 mins ago
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I completely agree, generally stationarity when referring to images draws on the distributions of the pixels themselves, but here it seems to be used based on the dependencies, treating the moving indices of the convolution to parametrize some from of process.
  $endgroup$
  – mshlis
  29 mins ago
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "658"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

Shirish Kulhari is a new contributor. Be nice, and check out our Code of Conduct.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fai.stackexchange.com%2fquestions%2f13683%2fwhat-is-the-meaning-of-stationarity-of-statistics-and-locality-of-pixel-depen%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

2

$begingroup$

locality of pixel dependencies probably means that neighboring pixels tend to be correlated, while faraway pixels are usually not correlated. This assumption is usually made in several image processing techniques (e.g. filters). Of course, the size and the shape of the neighborhood could vary, depending on the region of the image (or whatever), but, in practice, it is usually chosen to be fixed and rectangular (or squared).

stationarity of statistics probably means that the values of the pixels do not change over time.

With while their theoretically-best performance is likely to be only slightly worse the authors probably thought that, theoretically, CNNs are not as powerful as feedforward neural networks. However, both CNNs and FFNNs are universal function approximators (but, at the time, nobody probably had yet investigated seriously the theoretical powerfulness of CNNs).

share|improve this answer

answered 4 hours ago

nbronbro

5,72744 gold badges1515 silver badges3232 bronze badges

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I really think in this case stationarity isnt regarding time, but spatial location (given no time exists-- im assuming you meant over the dataset) I think theyre saying alot of local statistics are common across multiple areas of the image (therefore a single convolved filter can be a useful featurization of multiple fields rather than just one)
  $endgroup$
  – mshlis
  1 hour ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  @mshlis You might be right. This was my interpretation, given the usual meaning of stationarity (e.g. in reinforcement learning). However, if your interpretation is correct, then their terminology is highly confusing or misleading.
  $endgroup$
  – nbro
  59 mins ago
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I completely agree, generally stationarity when referring to images draws on the distributions of the pixels themselves, but here it seems to be used based on the dependencies, treating the moving indices of the convolution to parametrize some from of process.
  $endgroup$
  – mshlis
  29 mins ago
  

  
  
  
  

add a comment | 

2

$begingroup$

locality of pixel dependencies probably means that neighboring pixels tend to be correlated, while faraway pixels are usually not correlated. This assumption is usually made in several image processing techniques (e.g. filters). Of course, the size and the shape of the neighborhood could vary, depending on the region of the image (or whatever), but, in practice, it is usually chosen to be fixed and rectangular (or squared).

stationarity of statistics probably means that the values of the pixels do not change over time.

With while their theoretically-best performance is likely to be only slightly worse the authors probably thought that, theoretically, CNNs are not as powerful as feedforward neural networks. However, both CNNs and FFNNs are universal function approximators (but, at the time, nobody probably had yet investigated seriously the theoretical powerfulness of CNNs).

share|improve this answer

answered 4 hours ago

nbronbro

5,72744 gold badges1515 silver badges3232 bronze badges

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I really think in this case stationarity isnt regarding time, but spatial location (given no time exists-- im assuming you meant over the dataset) I think theyre saying alot of local statistics are common across multiple areas of the image (therefore a single convolved filter can be a useful featurization of multiple fields rather than just one)
  $endgroup$
  – mshlis
  1 hour ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  @mshlis You might be right. This was my interpretation, given the usual meaning of stationarity (e.g. in reinforcement learning). However, if your interpretation is correct, then their terminology is highly confusing or misleading.
  $endgroup$
  – nbro
  59 mins ago
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I completely agree, generally stationarity when referring to images draws on the distributions of the pixels themselves, but here it seems to be used based on the dependencies, treating the moving indices of the convolution to parametrize some from of process.
  $endgroup$
  – mshlis
  29 mins ago
  

  
  
  
  

add a comment | 

$begingroup$

locality of pixel dependencies probably means that neighboring pixels tend to be correlated, while faraway pixels are usually not correlated. This assumption is usually made in several image processing techniques (e.g. filters). Of course, the size and the shape of the neighborhood could vary, depending on the region of the image (or whatever), but, in practice, it is usually chosen to be fixed and rectangular (or squared).

stationarity of statistics probably means that the values of the pixels do not change over time.

With while their theoretically-best performance is likely to be only slightly worse the authors probably thought that, theoretically, CNNs are not as powerful as feedforward neural networks. However, both CNNs and FFNNs are universal function approximators (but, at the time, nobody probably had yet investigated seriously the theoretical powerfulness of CNNs).

share|improve this answer

answered 4 hours ago

nbronbro

5,72744 gold badges1515 silver badges3232 bronze badges

$endgroup$

share|improve this answer

answered 4 hours ago

nbronbro

5,72744 gold badges1515 silver badges3232 bronze badges

answered 4 hours ago

nbronbro

5,72744 gold badges1515 silver badges3232 bronze badges

answered 4 hours ago

nbronbro

5,72744 gold badges1515 silver badges3232 bronze badges